iPXE iso vs iPXE FOG

[Scott B]

I’m working on getting a batch of some order PCs reloaded and I can’t seem to get them to image via network boot with FOG. They hang at “iPXE initializing devices”. I can get them to image if I use the ipxe iso directly from ipxe’s website on a USB drive. Is there a way to update/change FOG to use what I’m guessing is an updated or just a different version of ipxe?

The server is running Ubuntu 18.x with FOG 1.5.9. Also with that, I downloaded the newest kernels from within the FOG web GUI. bzImage 5.10.19 and bzImage32 5.10.19. PCs in question use an Asus P8H61-M motherboard.

[Sebastian Roth]

@Scott-B It might be the iPXE version but could also be the devices behaving differently when booted via USB (iPXE) compared to PXE (also iPXE).

You can let FOG build iPXE form the very latest official iPXE code repo running those commands (assuming you have the FOG stuff extracted in /root/fogproject - just change the path to whereever you have it):

sudo -i
cd /root/fogproject/utils/FOGiPXE/
./buildipxe.sh
cd ../../bin/
./installfog.sh

[Scott B]

@sebastian-roth

Thanks for the tip. I didn’t know that was possible. I did perform those steps but I’m still seeing the issue.

[george1421]

@scott-b on the version you have on the cdrom that works, I’m suspecting its pretty old. When ipxe first starts it displays a banner message. At the very start of the banner page there will be a version number AND a hex code in square brackets. What are both codes. I haven’t had to do this before but we should be able to look up the commit code to find out how old the ipxe kernel is. Also look at the date created of the ipxe.efi file.

Usually the hang on initializing devices is related to a firmware bug in the bios. The Dell 7010s have this issue and will not start the ipxe kernel in uefi mode, where bios mode is just fine. With the 7010 being 10 years old, Dell has no motivation to fix the firmware.

If we can’t get iPXE to boot, no way… We can boot FOS Linux from a usb stick. There are some caveats with this method. Simple unicast imaging works. Its not currently possible to multicast image using a usb boot stick.

[Sebastian Roth]

@george1421 Yeah good point! @Scott-B Definitely check on the version numbers you see in the banner message. Should have said that before. If the compile went all well you should see g3efdb in the header. You can check the numbers here: https://github.com/ipxe/ipxe/commits/master (first part of the commit hash with a preceding g)

[Scott B]

@george1421 said in iPXE iso vs iPXE FOG:

@scott-b on the version you have on the cdrom that works, I’m suspecting its pretty old. When ipxe first starts it displays a banner message. At the very start of the banner page there will be a version number AND a hex code in square brackets. What are both codes. I haven’t had to do this before but we should be able to look up the commit code to find out how old the ipxe kernel is. Also look at the date created of the ipxe.efi file.

Here is a screenshot from booting FOG after updating iPXE.

Here is the screen when I boot from the iPXE USB.

Hard to see in the second picture, but they are the same.

[george1421]

@scott-b Ok I’m missing something both screen shots have the same version of iPXE and they both appear to pass the initializing devices section. The top picture appears to be a bios boot pxe boot the bottom doesn’t have enough detail.

So just to restate the version on the usb stick you downloaded appears to be the same version you complied with the fog buildpxe.sh schell script, but he fog built ipxe doesn’t boot and gets stuck at initng devices?

The only thing FOG adds special to the ipxe boot loader is a script that gets the dhcp address and then reconnect to the fog server to load default.ipxe. The remaining code base is all the same.

[Scott B]

@george1421

Sorry I didn’t explain that well. The first picture is not from the PC in question that started this thread. It’s from a PC that works fine with FOG. The PC in question will not get far enough to show that iPXE information from the network booting to FOG. It hangs with the "iPXE initializing devices” but no further information. I just wanted to post the screenshot to show the iso and fog are the same numbers.

[Scott B]

I just remade a new iPXE USB from the iso file and it reports iPXE 1.21.1+ (g85d17) but then comes up again with the (g3efd). This is different than the fog server. So now I’m confused.

Screenshot from the PC with the issue, booting from iPXE iso on USB from iPXE site.

[Sebastian Roth]

@scott-b said in iPXE iso vs iPXE FOG:

I just remade a new iPXE USB from the iso file and it reports iPXE 1.21.1+ (g85d17) but then comes up again with the (g3efd).

On the one hand side it’s ok you see two different versions when booting off the USB key because you have one version of iPXE on the USB key and that contacts the FOG server via TFTP to load and execute iPXE from there (line: tftp://...undionly.kpxe... ok in the picture). BUT the version string g85d17 is kinda strange because that’s a few commits earlier (from 14th of April): https://github.com/ipxe/ipxe/commits/master

There are different ways of booting off a USB key. You could even build an iPXE binary that has a script included which would not use the PXE boot information to load undionly.kpxe from the FOG server but directly chainload to default.ipxe skipping the secondary iPXE loading.

All in all that tells us that in your case the problematic hardware only seems to have an issue when iPXE is directly loaded via PXE. Even more strange I find. Have you tried using a different iPXE binary like snponly.efi yet?

[Scott B]

@sebastian-roth said in iPXE iso vs iPXE FOG:

Have you tried using a different iPXE binary like snponly.efi yet?

I have tried snponly.efi and realtek.efi with the same results.

[george1421]

@scott-b OK I’m a bit lost here where the problem is and if you have been able to overcome the issue.

So lets go back and restate the problem.

1. When I attempt to pxe boot the Asus P8H61-M motherboard using iPXE from FOG 1.5.9 in bios mode it fails at initializing devices… ?
2. When I use the same conditions but pxe boot in uefi mode it fails at initializing devices…
3. I can download ipxe.iso and burn it to a usb stick and it works
4. I use the FOG buildipxe.sh and rebuild ipxe boot loaders and it fails with initializing devices…
5. The version of iPXE from the ipxe.iso and what was build with buildipxe.sh is the same.

I just remade a new iPXE USB from the iso file and it reports iPXE 1.21.1+ (g85d17)

I don’t understand this statement how did you make a new iPXE USB from the iso. Just for clarity you did not compile the iso, but just used rufus or such to burn the iso to the usb?

Last question, you have only been able to boot ipxe completely with what version of iPXE (numbers in the square brackets)?

[Scott B]

@george1421 said in iPXE iso vs iPXE FOG:

Just for clarity you did not compile the iso, but just used rufus or such to burn the iso to the usb?
Last question, you have only been able to boot ipxe completely with what version of iPXE (numbers in the square brackets)?

Correct. I meant I used Rufus to burn the iso from ipxe’s website.

I was able to boot usb with (g85d17). But then that hands off to FOG which is using (g3efd)?

[george1421]

@scott-b said in iPXE iso vs iPXE FOG:

I was able to boot usb with (g85d17). But then that hands off to FOG which is using (g3efd)?

So 85d17 is from mid APR and the one from fog is much older.

Just so I understand when you ran the ./buildipxe.sh program that should have downloaded a current version of iPXE. So this should have been current as of the time you ran the script. I just ran the build script on my fog server and it did git the current ipxe build files.

[Edit] I did see that the buildipxe.sh script ONLY builds the efi version of the latest release. The bios versions did not get updated in my installer directory.

But then that hands off to FOG which is using (g3efd)?

This part has me a bit confused. How is this hand-off taking place. The iPXE from the cdrom should not know how unless you are typing in to chain load undionly.kpxe or ipxe.efi from the fog server directly from the boot usb drive (via the command line inside the iPXE shell). If you want to test chain loading chain load to your fog server and boot default.ipxe file. That will chain you into FOG installer.

[george1421]

@george1421 Following up on this a bit. After running the buildipxe.sh script I found that only the ipxe.efi files were updated in ../fogproject/packages/tftp directory. Digging into the script a bit more I found on my Cento 7 fog server there were some dependencies missing to properly build iPXE.

To get the complete iPXE files updated I needed to add the following centos packages.

yum install genisoimage  syslinux xz-devel -y

Then rerun the buildipxe.sh script. Once that was done I confirmed all of the iPXE files in ../fogproject/packages/tftp were correct.

FWIW the fog installer script ../bin/installfog.sh copies the files from ../fogproject/packages/tftp to the /tftpboot directory during installation.

[Scott B]

@george1421 said in iPXE iso vs iPXE FOG:

This part has me a bit confused. How is this hand-off taking place.

Here is a short video of booting from the ipxe iso. It will boot with (g85d17) then about the 7-second mark (g3efd) is loaded before successfully dropping me off at my fog menu.

https://youtu.be/wrkTdMxKQK4

[george1421]

@scott-b Ok I see and understand what its doing. Basically its booting from the flash drive then the newer version of iPXE is querying dhcp to get the next-server and boot-file name, then it chains to it. Handy but not what you need.

Lets circle back to what I found when I was rebuilding iPXE using the current verison (that appears where you need to be since that newer version of iPXE works)

Will you look into the git repo you downloaded to install FOG in ../fogproject/packages/tftp and see if all of the file dates are current (as of when you ran the buildipxe.sh scripts)? You can view the file dates with ls -la in that tftp directory.

[Scott B]

@george1421 said in iPXE iso vs iPXE FOG:

Will you look into the git repo you downloaded to install FOG in …/fogproject/packages/tftp and see if all of the file dates are current (as of when you ran the buildipxe.sh scripts)? You can view the file dates with ls -la in that tftp directory.

I ran through the build process again and here is the output. Most all the dates are from today.

drwxr-xr-x 5 root root    4096 Dec 11 18:20 .
drwxr-xr-x 8 root root    4096 Jun  1  2020 ..
drwxr-xr-x 4 root root    4096 Dec 11 18:20 10secdelay
drwxr-xr-x 2 root root    4096 Dec 11 18:20 arm64-efi
-rwxr-xr-x 1 root root     868 Jun  1  2020 boot.txt
drwxr-xr-x 2 root root    4096 Dec 11 18:20 i386-efi
-rw-r--r-- 1 root root  248896 Apr 28 13:43 intel.efi
-rw-r--r-- 1 root root   99833 Apr 28 13:42 intel.kkpxe
-rw-r--r-- 1 root root   99881 Apr 28 13:42 intel.kpxe
-rw-r--r-- 1 root root   99849 Apr 28 13:42 intel.pxe
-rw-r--r-- 1 root root 1047040 Apr 28 13:43 ipxe.efi
-rw-r--r-- 1 root root  892928 Apr 28 13:42 ipxe.iso
-rw-r--r-- 1 root root  359292 Apr 28 13:42 ipxe.kkpxe
-rw-r--r-- 1 root root  359340 Apr 28 13:42 ipxe.kpxe
-rw-r--r-- 1 root root  358889 Apr 28 13:42 ipxe.krn
-rw-r--r-- 1 root root  358889 Apr 28 13:42 ipxe.lkrn
-rw-r--r-- 1 root root  359438 Apr 28 13:42 ipxe.pxe
-rw-r--r-- 1 root root  376832 Apr 28 13:42 ipxe.usb
-rw-r--r-- 1 root root   26140 Jun  1  2020 memdisk
-rw-r--r-- 1 root root  278976 Apr 28 13:43 ncm--ecm--axge.efi
-rw-r--r-- 1 root root  247808 Apr 28 13:43 realtek.efi
-rw-r--r-- 1 root root  100572 Apr 28 13:42 realtek.kkpxe
-rw-r--r-- 1 root root  100620 Apr 28 13:42 realtek.kpxe
-rw-r--r-- 1 root root  100616 Apr 28 13:42 realtek.pxe
-rw-r--r-- 1 root root  247232 Apr 28 13:43 snp.efi
-rw-r--r-- 1 root root  247456 Apr 28 13:43 snponly.efi
-rw-r--r-- 1 root root   99305 Apr 28 13:42 undionly.kkpxe
-rw-r--r-- 1 root root   99353 Apr 28 13:42 undionly.kpxe
-rw-r--r-- 1 root root   99330 Apr 28 13:42 undionly.pxe

[george1421]

@scott-b So when you rerun the FOG installer and does the files in /tftpboot match the date of this list? If so you should be booting the latest version of iPXE

[Scott B]

@george1421 said in iPXE iso vs iPXE FOG:

@scott-b So when you rerun the FOG installer and does the files in /tftpboot match the date of this list? If so you should be booting the latest version of iPXE

All the files that are in /root/fogproject/packages/tftp/ are the same in /tftpboot for date and time.