Fog upgrade to 1.5 broke PXE installation

[george1421]

@csa It would be interesting to see what FOG is telling iPXE when this freeze happens. With a browser what output does this command give you? http://10.10.34.98/fog/service/ipxe/boot.php?mac=00:1e:c9:7a:36:48

Post the results here, please. Quite frankly the image shows a normal boot, what we don’t see is the rest of the ipxe commands that are being executed.

[Tom Elliott]

@csa I’m of the bet this may be similar to the other problems but it’s the ipxe binary being newer than what was in 1.4.

Could you try using the ipxe files from 1.4 and see if the machine starts working again? I really don’t think it’s anything with bootmenu generation.

[csa]

Morning @tom-elliott I pull the git repo and run the install program every time I upgrade. How can I safely role back to " ipxe files from 1.4"?

Thank you
Chakri

[Tom Elliott]

@csa You could just checkout the 1.4.0 tag:

git pull
git fetch --all --tags --prune
git checkout tags/1.4.0
cp packages/tftp/*.*pxe /tftpboot/
cp packages/tftp/*.efi /tftpboot/
git checkout dev-branch
git pull

That should do it, though I will admit I haven’t tested this all out.

[csa]

@tom-elliott The binaries are same size. There is no difference. Looks like they weren’t changed on the new version.

[Sebastian Roth]

@csa said in Fog upgrade to 1.5 broke PXE installation:

The binaries are same size

That’s a first sign but still does not mean that the binaries are the same. Please compare md5sums as well to make sure.

The other thing that comes to my mind… On that picture we see you try booting Ubuntu install kernel. Do you see the very same hang on init when booting into a FOG task like deploy or capture???

[Sebastian Roth]

@csa Any news on this?

[csa]

@sebastian-roth There is no difference. It still pauses at the installation.

Version 1.5 binaries

# ls -la /opt/BKUP/tftpboot/*.*pxe
-rw-r-xr-x 1 fog root    448 Sep 27 17:58 /opt/BKUP/tftpboot/default.ipxe
-rw-r-xr-x 1 fog root  94044 Sep 27 17:58 /opt/BKUP/tftpboot/intel.kkpxe
-rw-r-xr-x 1 fog root  94092 Sep 27 17:58 /opt/BKUP/tftpboot/intel.kpxe
-rw-r-xr-x 1 fog root  94121 Sep 27 17:58 /opt/BKUP/tftpboot/intel.pxe
-rw-r-xr-x 1 fog root 348364 Sep 27 17:58 /opt/BKUP/tftpboot/ipxe.kkpxe
-rw-r-xr-x 1 fog root 348412 Sep 27 17:58 /opt/BKUP/tftpboot/ipxe.kpxe
-rw-r-xr-x 1 fog root 348459 Sep 27 17:58 /opt/BKUP/tftpboot/ipxe.pxe
-rw-r-xr-x 1 fog root  94880 Sep 27 17:58 /opt/BKUP/tftpboot/realtek.kkpxe
-rw-r-xr-x 1 fog root  94928 Sep 27 17:58 /opt/BKUP/tftpboot/realtek.kpxe
-rw-r-xr-x 1 fog root  94892 Sep 27 17:58 /opt/BKUP/tftpboot/realtek.pxe
-rw-r-xr-x 1 fog root  93613 Sep 27 17:58 /opt/BKUP/tftpboot/undionly.kkpxe
-rw-r-xr-x 1 fog root  93661 Sep 27 17:58 /opt/BKUP/tftpboot/undionly.kpxe
-rw-r-xr-x 1 fog root  93645 Sep 27 17:58 /opt/BKUP/tftpboot/undionly.pxe

Version 1.4 binaries

~# ls -la /tftpboot/*.*pxe
-rw-r-xr-x 1 fog root    448 Sep 27 17:58 /tftpboot/default.ipxe
-rw-r-xr-x 1 fog root  95716 Oct 18 16:11 /tftpboot/intel.kkpxe
-rw-r-xr-x 1 fog root  95764 Oct 18 16:11 /tftpboot/intel.kpxe
-rw-r-xr-x 1 fog root  95765 Oct 18 16:11 /tftpboot/intel.pxe
-rw-r-xr-x 1 fog root 348865 Oct 18 16:11 /tftpboot/ipxe.kkpxe
-rw-r-xr-x 1 fog root 348913 Oct 18 16:11 /tftpboot/ipxe.kpxe
-rw-r-xr-x 1 fog root 348777 Oct 18 16:11 /tftpboot/ipxe.pxe
-rw-r-xr-x 1 fog root  96536 Oct 18 16:11 /tftpboot/realtek.kkpxe
-rw-r-xr-x 1 fog root  96584 Oct 18 16:11 /tftpboot/realtek.kpxe
-rw-r-xr-x 1 fog root  96605 Oct 18 16:11 /tftpboot/realtek.pxe
-rw-r-xr-x 1 fog root  95290 Oct 18 16:11 /tftpboot/undionly.kkpxe
-rw-r-xr-x 1 fog root  95338 Oct 18 16:11 /tftpboot/undionly.kpxe
-rw-r-xr-x 1 fog root  95380 Oct 18 16:11 /tftpboot/undionly.pxe

[Sebastian Roth]

@csa What do you mean by “There is no difference”?? Most if not all files differ in size! Why do you think they’re all the same?

As I already said days ago: On that picture we see you try booting Ubuntu install kernel. Do you see the very same hang on init when booting into a FOG task like deploy or capture???

[csa]

@sebastian-roth I meant to say the issue is still the same. It gets hung at the same spot. Fog task deploy or capture NO we don’t see any issues but those are images.
Thank you

[Sebastian Roth]

@csa So most probably it’s got something to do with the Ubuntu install you are trying to boot. Are you able to download those files using your web browser? http://10.10.34.98/fog/OS_IMAGES/ubuntu-16.04-DVD/install/netboot/ubuntu-installer/amd64/linux and http://10.10.34.98/fog/OS_IMAGES/ubuntu-16.04-DVD/install/netboot/ubuntu-installer/amd64/initrd.gz

Which versions of the binaries did you test? See Tom’s post where he suggested to checkout tags/1.4.0 and try those. Did you also checkout 1.3.0. Do they all have the hang?

[csa]

@Sebastian-Roth and @Tom-Elliott Thank you so much for your help. Backed-up my current images and cleanly installed from the latest GIT-repo 1.5. Coping back the same hooks worked again. Not sure what happened during the upgrade but fog is back in production now.

Thank you
Chakri