PXE Boot not working properly from Storage Node after Upgrade to 1.5.8
-
Hey guys,
This is an addition to the request https://forums.fogproject.org/topic/14230/pxe-boot-not-working-properly-from-storage-node, which was solved with 1.5.7.
I thought, that it would be good to just upgrade to 1.5.8, so that these things don’t happen again. However after upgrading both the master and storage, the PXE Boot fails yet again with the same error as yesterday. I’ve already tried --recreate-ca and --recreate-keys on the Storage Node.
Error & certstat:
https://imgur.com/a/OnJX221Thanks for any help
-
@Silv4n Please grab the install log files from
fogproject/bin/error_logs/of both nodes upload to a fileshare and post a link here.Be careful when using the
--recreate-caoption when running the installer. As soon as you use fog-client software this will break communication with all your clients as they are bound to the CA cert. -
@Sebastian-Roth No worries, there are no clients on the storage node, only on the master, but thanks for the info.
Error Logs: https://bit.ly/2vSpoBy
-
@Silv4n We need to make sure the CA cert file is the same in both placed on your FOG server:
md5sum /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/html/fog/management/other/ca.cert.pemAs well let’s check if the webserver certificate and key match:
openssl x509 -noout -modulus -in /var/www/fog/management/other/ssl/srvpublic.crt | openssl md5 openssl rsa -noout -modulus -in /opt/fog/snapins/ssl/.srvprivate.key | openssl md5And last but not least verify the webserver cert is being issued by the CA correctly:
openssl verify -verbose -CAfile /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/fog/management/other/ssl/srvpublic.crtPost output of all the commands here.
-
@Sebastian-Roth Ok, I’m gonna try that afterwards, kinda in the middle of something.
-
@Sebastian-Roth
First command:e15712aaee9359a90e94b46905018252 /opt/fog/snapins/ssl/CA/.fogCA.pem e15712aaee9359a90e94b46905018252 /var/www/html/fog/management/other/ca.cert.pemSecond command:
(stdin)= e1dc65877f9a55eb8c01744cf987bb50Third command:
(stdin)= e1dc65877f9a55eb8c01744cf987bb50Fourth command:
/var/www/fog/management/other/ssl/srvpublic.crt: OK -
@Silv4n All looks fine up to here. Can you please run
ls -al /tftpboot/and post output here. From the log output we see the installer ran at Feb 21 around 11 o’clock. Just want to see if the iPXE binaries have the same timestamp.As well I forgot one openssl command you shall run to see if the SHA1 fingerprint of the CA cert on the server matches the one we see in the picture:
openssl x509 -noout -fingerprint -sha1 -in /opt/fog/snapins/ssl/CA/.fogCA.pem openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ca.cert.pem -
@Sebastian-Roth
Command 1:drwxr-xr-x 6 fogproject root 4096 Feb 20 11:34 . drwxr-xr-x 26 root root 4096 Feb 21 11:01 .. drwxr-xr-x 4 fogproject root 4096 Feb 20 11:34 10secdelay drwxr-xr-x 2 fogproject root 4096 Feb 21 08:23 arm64-efi -rw-r-xr-x 1 fogproject root 868 Feb 21 11:02 boot.txt -rw-r-xr-x 1 fogproject root 457 Feb 21 11:02 default.ipxe drwxr-xr-x 2 fogproject root 4096 Feb 20 11:34 i386-efi -rw-r-xr-x 1 fogproject root 227424 Feb 21 11:02 intel.efi -rw-r-xr-x 1 fogproject root 99123 Feb 21 11:02 intel.kkpxe -rw-r-xr-x 1 fogproject root 99171 Feb 21 11:02 intel.kpxe -rw-r-xr-x 1 fogproject root 99146 Feb 21 11:02 intel.pxe -rw-r-xr-x 1 fogproject root 1007360 Feb 21 11:02 ipxe.efi -rw-r-xr-x 1 fogproject root 876544 Feb 21 11:02 ipxe.iso -rw-r-xr-x 1 fogproject root 358066 Feb 21 11:02 ipxe.kkpxe -rw-r-xr-x 1 fogproject root 358114 Feb 21 11:02 ipxe.kpxe -rw-r-xr-x 1 fogproject root 357700 Feb 21 11:02 ipxe.krn -rw-r-xr-x 1 fogproject root 357700 Feb 21 11:02 ipxe.lkrn -rw-r-xr-x 1 fogproject root 358328 Feb 21 11:02 ipxe.pxe -rw-r-xr-x 1 fogproject root 1409024 Feb 21 11:02 ipxe.usb -rw-r-xr-x 1 fogproject root 123448 Feb 20 13:24 ldlinux.c32 -rw-r-xr-x 1 fogproject root 187820 Feb 20 13:24 libcom32.c32 -rw-r-xr-x 1 fogproject root 26468 Feb 20 13:24 libutil.c32 -rw-r-xr-x 1 fogproject root 26140 Feb 21 11:02 memdisk -rw-r-xr-x 1 fogproject root 29208 Feb 20 13:24 menu.c32 -rw-r-xr-x 1 fogproject root 252768 Feb 21 11:02 ncm--ecm--axge.efi -rw-r-xr-x 1 fogproject root 43210 Feb 20 13:24 pxelinux.0.old drwxr-xr-x 2 fogproject root 4096 Feb 18 08:17 pxelinux.cfg -rw-r-xr-x 1 fogproject root 226272 Feb 21 11:02 realtek.efi -rw-r-xr-x 1 fogproject root 99950 Feb 21 11:02 realtek.kkpxe -rw-r-xr-x 1 fogproject root 99998 Feb 21 11:02 realtek.kpxe -rw-r-xr-x 1 fogproject root 99968 Feb 21 11:02 realtek.pxe -rw-r-xr-x 1 fogproject root 225696 Feb 21 11:02 snp.efi -rw-r-xr-x 1 fogproject root 225952 Feb 21 11:02 snponly.efi -rw-r-xr-x 1 fogproject root 98645 Feb 21 11:02 undionly.kkpxe -rw-r-xr-x 1 fogproject root 98693 Feb 21 11:02 undionly.kpxe -rw-r-xr-x 1 fogproject root 98696 Feb 21 11:02 undionly.pxe -rw-r-xr-x 1 fogproject root 29728 Feb 20 13:24 vesamenu.c32Command 2:
SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6FCommand 3:
SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6F -
@Silv4n Still all good. We shall find it soon I am sure. Try these commands:
openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | head -
@Silv4n And here is one more command:
echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | openssl x509 -noout -fingerprint -
fogadmin@v-fogsrv02:~$ openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93 fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | head depth=1 CN = FOG Server CA verify return:1 depth=0 CN = 10.144.1.22 verify return:1 DONE CONNECTED(00000005) --- Certificate chain 0 s:CN = 10.144.1.22 i:CN = FOG Server CA 1 s:CN = FOG Server CA i:CN = FOG Server CA --- Server certificate -----BEGIN CERTIFICATE----- fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | openssl x509 -noout -fingerprint depth=1 CN = FOG Server CA verify return:1 depth=0 CN = 10.144.1.22 verify return:1 DONE SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93 -
@Silv4n This is really strange. All the certificates seem perfectly fine and match the fingerprints we see in the picture you posted initially. I just did a fresh clean install here and it worked out of the box. Though this is a master server only. Let me try adding a storage node and see if that makes a difference.
-
@Sebastian-Roth That’s the storage node
-
@Silv4n Let’s switch over to chat (chat bubble in the top right corner).
-
@Sebastian-Roth unfortunatly the can’t display all of it: https://imgur.com/a/OKdQzwh
-
@Silv4n Ok, unfortunately not of much help yet. Please recompile but leave out the
tls,in DEBUG parameter…make EMBED=ipxescript DEBUG=x509,validator bin/undionly.kpxe CERT=/opt/fog/snapins/ssl/CA/.fogCA.pem TRUST=/opt/fog/snapins/ssl/CA/.fogCA.pem cp bin/undionly.kpxe /tftpboot -
-
After some extended research I figured out this was caused by the build script not re-generating the trusted root part of the code compiled into the iPXE binaries. It’s really easy to fix and I pushed a fix to both
dev-branchandworking-1.6so we hopefully never run into this again.cd path/to/fogproject/bin/ touch ../../ipxe/src/crypto/rootcert.c rm /tftpboot/undionly.kkpxe ./installfog.shOn that way I learned a couple of things about iPXE booting over HTTPS and so I hope we can find most upcoming issues more quickly from now on.