PXE Boot not working properly from Storage Node after Upgrade to 1.5.8

[Silv4n]

@Sebastian-Roth
Command 1:

drwxr-xr-x  6 fogproject root    4096 Feb 20 11:34 .
drwxr-xr-x 26 root       root    4096 Feb 21 11:01 ..
drwxr-xr-x  4 fogproject root    4096 Feb 20 11:34 10secdelay
drwxr-xr-x  2 fogproject root    4096 Feb 21 08:23 arm64-efi
-rw-r-xr-x  1 fogproject root     868 Feb 21 11:02 boot.txt
-rw-r-xr-x  1 fogproject root     457 Feb 21 11:02 default.ipxe
drwxr-xr-x  2 fogproject root    4096 Feb 20 11:34 i386-efi
-rw-r-xr-x  1 fogproject root  227424 Feb 21 11:02 intel.efi
-rw-r-xr-x  1 fogproject root   99123 Feb 21 11:02 intel.kkpxe
-rw-r-xr-x  1 fogproject root   99171 Feb 21 11:02 intel.kpxe
-rw-r-xr-x  1 fogproject root   99146 Feb 21 11:02 intel.pxe
-rw-r-xr-x  1 fogproject root 1007360 Feb 21 11:02 ipxe.efi
-rw-r-xr-x  1 fogproject root  876544 Feb 21 11:02 ipxe.iso
-rw-r-xr-x  1 fogproject root  358066 Feb 21 11:02 ipxe.kkpxe
-rw-r-xr-x  1 fogproject root  358114 Feb 21 11:02 ipxe.kpxe
-rw-r-xr-x  1 fogproject root  357700 Feb 21 11:02 ipxe.krn
-rw-r-xr-x  1 fogproject root  357700 Feb 21 11:02 ipxe.lkrn
-rw-r-xr-x  1 fogproject root  358328 Feb 21 11:02 ipxe.pxe
-rw-r-xr-x  1 fogproject root 1409024 Feb 21 11:02 ipxe.usb
-rw-r-xr-x  1 fogproject root  123448 Feb 20 13:24 ldlinux.c32
-rw-r-xr-x  1 fogproject root  187820 Feb 20 13:24 libcom32.c32
-rw-r-xr-x  1 fogproject root   26468 Feb 20 13:24 libutil.c32
-rw-r-xr-x  1 fogproject root   26140 Feb 21 11:02 memdisk
-rw-r-xr-x  1 fogproject root   29208 Feb 20 13:24 menu.c32
-rw-r-xr-x  1 fogproject root  252768 Feb 21 11:02 ncm--ecm--axge.efi
-rw-r-xr-x  1 fogproject root   43210 Feb 20 13:24 pxelinux.0.old
drwxr-xr-x  2 fogproject root    4096 Feb 18 08:17 pxelinux.cfg
-rw-r-xr-x  1 fogproject root  226272 Feb 21 11:02 realtek.efi
-rw-r-xr-x  1 fogproject root   99950 Feb 21 11:02 realtek.kkpxe
-rw-r-xr-x  1 fogproject root   99998 Feb 21 11:02 realtek.kpxe
-rw-r-xr-x  1 fogproject root   99968 Feb 21 11:02 realtek.pxe
-rw-r-xr-x  1 fogproject root  225696 Feb 21 11:02 snp.efi
-rw-r-xr-x  1 fogproject root  225952 Feb 21 11:02 snponly.efi
-rw-r-xr-x  1 fogproject root   98645 Feb 21 11:02 undionly.kkpxe
-rw-r-xr-x  1 fogproject root   98693 Feb 21 11:02 undionly.kpxe
-rw-r-xr-x  1 fogproject root   98696 Feb 21 11:02 undionly.pxe
-rw-r-xr-x  1 fogproject root   29728 Feb 20 13:24 vesamenu.c32

Command 2:

SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6F

Command 3:

SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6F

[Sebastian Roth]

@Silv4n Still all good. We shall find it soon I am sure. Try these commands:

openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt
echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | head

[Sebastian Roth]

@Silv4n And here is one more command:

echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | openssl x509  -noout -fingerprint

[Silv4n]

@Sebastian-Roth

fogadmin@v-fogsrv02:~$ openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt
SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93
fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | head
depth=1 CN = FOG Server CA
verify return:1
depth=0 CN = 10.144.1.22
verify return:1
DONE
CONNECTED(00000005)
---
Certificate chain
 0 s:CN = 10.144.1.22
   i:CN = FOG Server CA
 1 s:CN = FOG Server CA
   i:CN = FOG Server CA
---
Server certificate
-----BEGIN CERTIFICATE-----

fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | openssl x509  -noout -fingerprint
depth=1 CN = FOG Server CA
verify return:1
depth=0 CN = 10.144.1.22
verify return:1
DONE
SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93

[Sebastian Roth]

@Silv4n This is really strange. All the certificates seem perfectly fine and match the fingerprints we see in the picture you posted initially. I just did a fresh clean install here and it worked out of the box. Though this is a master server only. Let me try adding a storage node and see if that makes a difference.

[Silv4n]

@Sebastian-Roth That’s the storage node

[Sebastian Roth]

@Silv4n Let’s switch over to chat (chat bubble in the top right corner).

[Silv4n]

@Sebastian-Roth unfortunatly the can’t display all of it: https://imgur.com/a/OKdQzwh

[Sebastian Roth]

@Silv4n Ok, unfortunately not of much help yet. Please recompile but leave out the tls, in DEBUG parameter…

make EMBED=ipxescript DEBUG=x509,validator bin/undionly.kpxe CERT=/opt/fog/snapins/ssl/CA/.fogCA.pem TRUST=/opt/fog/snapins/ssl/CA/.fogCA.pem
cp bin/undionly.kpxe /tftpboot

[Silv4n]

@Sebastian-Roth https://imgur.com/a/j0WJInw

[Sebastian Roth]

After some extended research I figured out this was caused by the build script not re-generating the trusted root part of the code compiled into the iPXE binaries. It’s really easy to fix and I pushed a fix to both dev-branch and working-1.6 so we hopefully never run into this again.

cd path/to/fogproject/bin/
touch ../../ipxe/src/crypto/rootcert.c
rm /tftpboot/undionly.kkpxe
./installfog.sh

On that way I learned a couple of things about iPXE booting over HTTPS and so I hope we can find most upcoming issues more quickly from now on.