PXE Boot not working properly from Storage Node after Upgrade to 1.5.8

[Silv4n]

Hey guys,

This is an addition to the request https://forums.fogproject.org/topic/14230/pxe-boot-not-working-properly-from-storage-node, which was solved with 1.5.7.

I thought, that it would be good to just upgrade to 1.5.8, so that these things don’t happen again. However after upgrading both the master and storage, the PXE Boot fails yet again with the same error as yesterday. I’ve already tried --recreate-ca and --recreate-keys on the Storage Node.

Error & certstat:
https://imgur.com/a/OnJX221

Thanks for any help

[Sebastian Roth]

@Silv4n Please grab the install log files from fogproject/bin/error_logs/ of both nodes upload to a fileshare and post a link here.

Be careful when using the --recreate-ca option when running the installer. As soon as you use fog-client software this will break communication with all your clients as they are bound to the CA cert.

[Silv4n]

@Sebastian-Roth No worries, there are no clients on the storage node, only on the master, but thanks for the info.

Error Logs: https://bit.ly/2vSpoBy

[Sebastian Roth]

@Silv4n We need to make sure the CA cert file is the same in both placed on your FOG server:

md5sum /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/html/fog/management/other/ca.cert.pem

As well let’s check if the webserver certificate and key match:

openssl x509 -noout -modulus -in /var/www/fog/management/other/ssl/srvpublic.crt | openssl md5
openssl rsa -noout -modulus -in /opt/fog/snapins/ssl/.srvprivate.key | openssl md5

And last but not least verify the webserver cert is being issued by the CA correctly:

openssl verify -verbose -CAfile /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/fog/management/other/ssl/srvpublic.crt

Post output of all the commands here.

[Silv4n]

@Sebastian-Roth Ok, I’m gonna try that afterwards, kinda in the middle of something.

[Silv4n]

@Sebastian-Roth
First command:

e15712aaee9359a90e94b46905018252  /opt/fog/snapins/ssl/CA/.fogCA.pem
e15712aaee9359a90e94b46905018252  /var/www/html/fog/management/other/ca.cert.pem

Second command:

(stdin)= e1dc65877f9a55eb8c01744cf987bb50

Third command:

(stdin)= e1dc65877f9a55eb8c01744cf987bb50

Fourth command:

/var/www/fog/management/other/ssl/srvpublic.crt: OK

[Sebastian Roth]

@Silv4n All looks fine up to here. Can you please run ls -al /tftpboot/ and post output here. From the log output we see the installer ran at Feb 21 around 11 o’clock. Just want to see if the iPXE binaries have the same timestamp.

As well I forgot one openssl command you shall run to see if the SHA1 fingerprint of the CA cert on the server matches the one we see in the picture:

openssl x509 -noout -fingerprint -sha1 -in /opt/fog/snapins/ssl/CA/.fogCA.pem
openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ca.cert.pem

[Silv4n]

@Sebastian-Roth
Command 1:

drwxr-xr-x  6 fogproject root    4096 Feb 20 11:34 .
drwxr-xr-x 26 root       root    4096 Feb 21 11:01 ..
drwxr-xr-x  4 fogproject root    4096 Feb 20 11:34 10secdelay
drwxr-xr-x  2 fogproject root    4096 Feb 21 08:23 arm64-efi
-rw-r-xr-x  1 fogproject root     868 Feb 21 11:02 boot.txt
-rw-r-xr-x  1 fogproject root     457 Feb 21 11:02 default.ipxe
drwxr-xr-x  2 fogproject root    4096 Feb 20 11:34 i386-efi
-rw-r-xr-x  1 fogproject root  227424 Feb 21 11:02 intel.efi
-rw-r-xr-x  1 fogproject root   99123 Feb 21 11:02 intel.kkpxe
-rw-r-xr-x  1 fogproject root   99171 Feb 21 11:02 intel.kpxe
-rw-r-xr-x  1 fogproject root   99146 Feb 21 11:02 intel.pxe
-rw-r-xr-x  1 fogproject root 1007360 Feb 21 11:02 ipxe.efi
-rw-r-xr-x  1 fogproject root  876544 Feb 21 11:02 ipxe.iso
-rw-r-xr-x  1 fogproject root  358066 Feb 21 11:02 ipxe.kkpxe
-rw-r-xr-x  1 fogproject root  358114 Feb 21 11:02 ipxe.kpxe
-rw-r-xr-x  1 fogproject root  357700 Feb 21 11:02 ipxe.krn
-rw-r-xr-x  1 fogproject root  357700 Feb 21 11:02 ipxe.lkrn
-rw-r-xr-x  1 fogproject root  358328 Feb 21 11:02 ipxe.pxe
-rw-r-xr-x  1 fogproject root 1409024 Feb 21 11:02 ipxe.usb
-rw-r-xr-x  1 fogproject root  123448 Feb 20 13:24 ldlinux.c32
-rw-r-xr-x  1 fogproject root  187820 Feb 20 13:24 libcom32.c32
-rw-r-xr-x  1 fogproject root   26468 Feb 20 13:24 libutil.c32
-rw-r-xr-x  1 fogproject root   26140 Feb 21 11:02 memdisk
-rw-r-xr-x  1 fogproject root   29208 Feb 20 13:24 menu.c32
-rw-r-xr-x  1 fogproject root  252768 Feb 21 11:02 ncm--ecm--axge.efi
-rw-r-xr-x  1 fogproject root   43210 Feb 20 13:24 pxelinux.0.old
drwxr-xr-x  2 fogproject root    4096 Feb 18 08:17 pxelinux.cfg
-rw-r-xr-x  1 fogproject root  226272 Feb 21 11:02 realtek.efi
-rw-r-xr-x  1 fogproject root   99950 Feb 21 11:02 realtek.kkpxe
-rw-r-xr-x  1 fogproject root   99998 Feb 21 11:02 realtek.kpxe
-rw-r-xr-x  1 fogproject root   99968 Feb 21 11:02 realtek.pxe
-rw-r-xr-x  1 fogproject root  225696 Feb 21 11:02 snp.efi
-rw-r-xr-x  1 fogproject root  225952 Feb 21 11:02 snponly.efi
-rw-r-xr-x  1 fogproject root   98645 Feb 21 11:02 undionly.kkpxe
-rw-r-xr-x  1 fogproject root   98693 Feb 21 11:02 undionly.kpxe
-rw-r-xr-x  1 fogproject root   98696 Feb 21 11:02 undionly.pxe
-rw-r-xr-x  1 fogproject root   29728 Feb 20 13:24 vesamenu.c32

Command 2:

SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6F

Command 3:

SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6F

[Sebastian Roth]

@Silv4n Still all good. We shall find it soon I am sure. Try these commands:

openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt
echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | head

[Sebastian Roth]

@Silv4n And here is one more command:

echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | openssl x509  -noout -fingerprint

[Silv4n]

@Sebastian-Roth

fogadmin@v-fogsrv02:~$ openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt
SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93
fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | head
depth=1 CN = FOG Server CA
verify return:1
depth=0 CN = 10.144.1.22
verify return:1
DONE
CONNECTED(00000005)
---
Certificate chain
 0 s:CN = 10.144.1.22
   i:CN = FOG Server CA
 1 s:CN = FOG Server CA
   i:CN = FOG Server CA
---
Server certificate
-----BEGIN CERTIFICATE-----

fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | openssl x509  -noout -fingerprint
depth=1 CN = FOG Server CA
verify return:1
depth=0 CN = 10.144.1.22
verify return:1
DONE
SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93

[Sebastian Roth]

@Silv4n This is really strange. All the certificates seem perfectly fine and match the fingerprints we see in the picture you posted initially. I just did a fresh clean install here and it worked out of the box. Though this is a master server only. Let me try adding a storage node and see if that makes a difference.

[Silv4n]

@Sebastian-Roth That’s the storage node

[Sebastian Roth]

@Silv4n Let’s switch over to chat (chat bubble in the top right corner).

[Silv4n]

@Sebastian-Roth unfortunatly the can’t display all of it: https://imgur.com/a/OKdQzwh

[Sebastian Roth]

@Silv4n Ok, unfortunately not of much help yet. Please recompile but leave out the tls, in DEBUG parameter…

make EMBED=ipxescript DEBUG=x509,validator bin/undionly.kpxe CERT=/opt/fog/snapins/ssl/CA/.fogCA.pem TRUST=/opt/fog/snapins/ssl/CA/.fogCA.pem
cp bin/undionly.kpxe /tftpboot

[Silv4n]

@Sebastian-Roth https://imgur.com/a/j0WJInw

[Sebastian Roth]

After some extended research I figured out this was caused by the build script not re-generating the trusted root part of the code compiled into the iPXE binaries. It’s really easy to fix and I pushed a fix to both dev-branch and working-1.6 so we hopefully never run into this again.

cd path/to/fogproject/bin/
touch ../../ipxe/src/crypto/rootcert.c
rm /tftpboot/undionly.kkpxe
./installfog.sh

On that way I learned a couple of things about iPXE booting over HTTPS and so I hope we can find most upcoming issues more quickly from now on.