Fog node - exec format error, chainloading failed

[mtanigawa]

Server:
• FOG Version: 1.5.6
• OS: Ubuntu 16.04.6 LTS

We are looking to move over to UEFI next summer and have started testing. Our initial tests over a break were successful and we were able to capture and deploy via UEFI boot using FOG 1.5.5.

Sometime between then and a couple weeks ago, we are no longer able to capture or deploy. No changes have been made between the initial and failed tests.

We are using ipxe.efi as the boot file. When booting the client, we get the error:

...
tftp://fog-node-ip/default.ipxe... ok
http://fog-master-ip/fog/service/ipxe/boot.php... ok
bzimage... ok
Could not select: Exec format error (http://ipxe.org/2e008081)
Could not boot: Exec format error (http://ipxe.org/2e008081)
Could not boot: Exec format error (http://ipxe.org/2e008081)
Chainloding failed, hit 's' for the iPXE shell; reboot in ... seconds

When again taking a further look at this, fog was updated to 1.5.6 - hoping the update would resolve this issue. After updating, we have again tried to capture / deploy and the same error (above) is displayed on the client.

As a test, I have build a fresh version of fog (with no nodes - v1.5.5) and when pointing the test server, we are able to capture an image. When deploying, the client goes right to the refind_efi menu and not the fog menu…

I have gone through and checked permissions, ownership and settings between our production fog and our test fog. I have not been able to find anything that would point me in any direction to figure out what may be going on.

I have gone through the BIOS settings on the client (secure boot off, TPM disabled, etc.). I have also tried different variations of this based on other discussions on this forum for the errors received. Unfortunately, many of the other posts have not helped to move me in the right direction.

I would like to see if there are other things / ideas I am able to check / try in the hopes of resolving this. Please let me know should additional information be needed.

Thank you for your time.

[george1421]

Will you confirm if this post resolves your issue: https://forums.fogproject.org/topic/10006/ubuntu-is-fog-s-enemy

[mtanigawa]

@george1421

Thanks for the follow up. Yes, it is one of the things I tried.

As a follow up to this, I have restored fog from a previous backup (Jan) and all now works - tested both pxe and uefi boots - capture and restore.

I will leave this as is, as it now works. I did save a damaged config export to play with later if needed.

Thank you.

[Sebastian Roth]

@mtanigawa said:

http://fog-master-ip/fog/service/ipxe/boot.php… ok
bzimage… ok
Could not select: Exec format error (http://ipxe.org/2e008081)

The error means that iPXE is not able to load the FOS system. If I get this right here iPXE is able to download the kernel part (bzImage... ok but can’t select and execute it for chainloading.

Either the kernel (or init) is damaged. Possibly just re-download those manually to see. Ask here if you need the URLs. Or the generated boot menu is corrupted. To check that open http://fog-master-ip/fog/service/ipxe/boot.php?mac=aa:bb:cc:dd:ee:ff in the browser (put in FOG server IP and client MAC address!). Post the full text output you get in the browser here in the forums.

[Sebastian Roth]

@mtanigawa Ahhhhhhhhhhhh, I might have spotted it. If this is not a typo in your post (bzimage… ok) then I suppose you just have the wrong Host Kernel setting (for this particular client or in generel settings for UEFI clients). Linux is case sensitive and bzimage is not same as bzImage (the default name in FOG).

If that’s the case I was wrong with my first assumption that iPXE was able to download the kernel binary!

[mtanigawa]

@Sebastian-Roth

Thanks for your additional details.

Initially when checking into this error, I did replace the bzimage w/ a clean on from the working test server. I assume that the bzimage was again replaced when I updated from 1.5.5 to 1.5.6. It may have been my typo on the bzimage text.

I will load the corrupted backup onto a test box and see what comes out of it and check the URL posted.

Thank you.

[Sebastian Roth]

@mtanigawa Shall we keep this issue open. Will you get to look into this over the next days?

[mtanigawa]

@Sebastian-Roth

Yes, if you wouldn’t mind please. I will prep a box tomorrow with default installs and restore the backup. Will then run through and let you know what I come up with … or if it all decides to just work.

Thank you.

[Gordon Taylor]

i think this is the same error as mine that started yesterday and it happened after i did a apt-get upgrade… i noticed at the end of the upgrade i has these lines:

NOTICE: Not enabling PHP 7.1 FPM by default.
NOTICE: To enable PHP 7.1 FPM in Apache2 do:
NOTICE: a2enmod proxy_fcgi setenvif
NOTICE: a2enconf php7.1-fpm
NOTICE: You are seeing this message because you have apache2 package installed.

ive just switched my fog server off as its stoping the workstations to boot and i cant stay late to troubleshoot it today… but i can tomorrow…

[Gordon Taylor]

@Gordon-Taylor ignore me… ive just reinstalled fog and mine is fine now… the mysql bug as mentioned at the begining of the thread

[george1421]

@Gordon-Taylor said in Fog node - exec format error, chainloading failed:

the mysql bug as mentioned at the begining of the thread

Hopefully this issue will be fixed in 1.5.7 or 1.5.8 where we should not have this issue moving forward, right @Developers

[Sebastian Roth]

@Gordon-Taylor Am I on the wrong track here? I can’t see your issue being related to the one posted here in any way. Here we are very likely talking about a miss-configuration (typo in bzImage name). Did you get the exact same error message “Could not select: Exec format error”??

@george1421 @Gordon-Taylor What mysql bug are we talking about here?! Do you mean using an empty mysql root account password in FOG?

[george1421]

@Sebastian-Roth said in Fog node - exec format error, chainloading failed:

Do you mean using an empty mysql root account password in FOG?

Yep, that one. We can talk about it off line.

[Sebastian Roth]

@george1421 Can we move this discussion to another thread or even better open an issue for it so it won’t get lost in the depths of the forum??

[mtanigawa]

@Sebastian-Roth

I have built a vanilla box, installed fog and restored my configuration … and it all worked! I was able to boot via pxe and uefi on the first try of each.

I will chalk this up to both the blank mysql password and a miss named bzImage file. As it is all working with the “bad” backup config, I will ask that this ticket be closed. One day, if I’m bored enough, I’ll try another restore again and see if it will fail.

Thank you for all the help! It is greatly appreciated.