SOLVED PXE Boot not working properly from Storage Node after Upgrade to 1.5.8

  • Hey guys,

    This is an addition to the request, which was solved with 1.5.7.

    I thought, that it would be good to just upgrade to 1.5.8, so that these things don’t happen again. However after upgrading both the master and storage, the PXE Boot fails yet again with the same error as yesterday. I’ve already tried --recreate-ca and --recreate-keys on the Storage Node.

    Error & certstat:

    Thanks for any help

  • Moderator

    After some extended research I figured out this was caused by the build script not re-generating the trusted root part of the code compiled into the iPXE binaries. It’s really easy to fix and I pushed a fix to both dev-branch and working-1.6 so we hopefully never run into this again.

    cd path/to/fogproject/bin/
    touch ../../ipxe/src/crypto/rootcert.c
    rm /tftpboot/undionly.kkpxe

    On that way I learned a couple of things about iPXE booting over HTTPS and so I hope we can find most upcoming issues more quickly from now on.

  • Moderator

    @Silv4n Ok, unfortunately not of much help yet. Please recompile but leave out the tls, in DEBUG parameter…

    make EMBED=ipxescript DEBUG=x509,validator bin/undionly.kpxe CERT=/opt/fog/snapins/ssl/CA/.fogCA.pem TRUST=/opt/fog/snapins/ssl/CA/.fogCA.pem
    cp bin/undionly.kpxe /tftpboot

  • @Sebastian-Roth unfortunatly the can’t display all of it:

  • Moderator

    @Silv4n Let’s switch over to chat (chat bubble in the top right corner).

  • @Sebastian-Roth That’s the storage node

  • Moderator

    @Silv4n This is really strange. All the certificates seem perfectly fine and match the fingerprints we see in the picture you posted initially. I just did a fresh clean install here and it worked out of the box. Though this is a master server only. Let me try adding a storage node and see if that makes a difference.

  • @Sebastian-Roth

    fogadmin@v-fogsrv02:~$ openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt
    SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93
    fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect | head
    depth=1 CN = FOG Server CA
    verify return:1
    depth=0 CN =
    verify return:1
    Certificate chain
     0 s:CN =
       i:CN = FOG Server CA
     1 s:CN = FOG Server CA
       i:CN = FOG Server CA
    Server certificate
    fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect | openssl x509  -noout -fingerprint
    depth=1 CN = FOG Server CA
    verify return:1
    depth=0 CN =
    verify return:1
    SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93
  • Moderator

    @Silv4n And here is one more command:

    echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect | openssl x509  -noout -fingerprint
  • Moderator

    @Silv4n Still all good. We shall find it soon I am sure. Try these commands:

    openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt
    echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect | head

  • @Sebastian-Roth
    Command 1:

    drwxr-xr-x  6 fogproject root    4096 Feb 20 11:34 .
    drwxr-xr-x 26 root       root    4096 Feb 21 11:01 ..
    drwxr-xr-x  4 fogproject root    4096 Feb 20 11:34 10secdelay
    drwxr-xr-x  2 fogproject root    4096 Feb 21 08:23 arm64-efi
    -rw-r-xr-x  1 fogproject root     868 Feb 21 11:02 boot.txt
    -rw-r-xr-x  1 fogproject root     457 Feb 21 11:02 default.ipxe
    drwxr-xr-x  2 fogproject root    4096 Feb 20 11:34 i386-efi
    -rw-r-xr-x  1 fogproject root  227424 Feb 21 11:02 intel.efi
    -rw-r-xr-x  1 fogproject root   99123 Feb 21 11:02 intel.kkpxe
    -rw-r-xr-x  1 fogproject root   99171 Feb 21 11:02 intel.kpxe
    -rw-r-xr-x  1 fogproject root   99146 Feb 21 11:02 intel.pxe
    -rw-r-xr-x  1 fogproject root 1007360 Feb 21 11:02 ipxe.efi
    -rw-r-xr-x  1 fogproject root  876544 Feb 21 11:02 ipxe.iso
    -rw-r-xr-x  1 fogproject root  358066 Feb 21 11:02 ipxe.kkpxe
    -rw-r-xr-x  1 fogproject root  358114 Feb 21 11:02 ipxe.kpxe
    -rw-r-xr-x  1 fogproject root  357700 Feb 21 11:02 ipxe.krn
    -rw-r-xr-x  1 fogproject root  357700 Feb 21 11:02 ipxe.lkrn
    -rw-r-xr-x  1 fogproject root  358328 Feb 21 11:02 ipxe.pxe
    -rw-r-xr-x  1 fogproject root 1409024 Feb 21 11:02 ipxe.usb
    -rw-r-xr-x  1 fogproject root  123448 Feb 20 13:24 ldlinux.c32
    -rw-r-xr-x  1 fogproject root  187820 Feb 20 13:24 libcom32.c32
    -rw-r-xr-x  1 fogproject root   26468 Feb 20 13:24 libutil.c32
    -rw-r-xr-x  1 fogproject root   26140 Feb 21 11:02 memdisk
    -rw-r-xr-x  1 fogproject root   29208 Feb 20 13:24 menu.c32
    -rw-r-xr-x  1 fogproject root  252768 Feb 21 11:02 ncm--ecm--axge.efi
    -rw-r-xr-x  1 fogproject root   43210 Feb 20 13:24 pxelinux.0.old
    drwxr-xr-x  2 fogproject root    4096 Feb 18 08:17 pxelinux.cfg
    -rw-r-xr-x  1 fogproject root  226272 Feb 21 11:02 realtek.efi
    -rw-r-xr-x  1 fogproject root   99950 Feb 21 11:02 realtek.kkpxe
    -rw-r-xr-x  1 fogproject root   99998 Feb 21 11:02 realtek.kpxe
    -rw-r-xr-x  1 fogproject root   99968 Feb 21 11:02 realtek.pxe
    -rw-r-xr-x  1 fogproject root  225696 Feb 21 11:02 snp.efi
    -rw-r-xr-x  1 fogproject root  225952 Feb 21 11:02 snponly.efi
    -rw-r-xr-x  1 fogproject root   98645 Feb 21 11:02 undionly.kkpxe
    -rw-r-xr-x  1 fogproject root   98693 Feb 21 11:02 undionly.kpxe
    -rw-r-xr-x  1 fogproject root   98696 Feb 21 11:02 undionly.pxe
    -rw-r-xr-x  1 fogproject root   29728 Feb 20 13:24 vesamenu.c32

    Command 2:

    SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6F

    Command 3:

    SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6F
  • Moderator

    @Silv4n All looks fine up to here. Can you please run ls -al /tftpboot/ and post output here. From the log output we see the installer ran at Feb 21 around 11 o’clock. Just want to see if the iPXE binaries have the same timestamp.

    As well I forgot one openssl command you shall run to see if the SHA1 fingerprint of the CA cert on the server matches the one we see in the picture:

    openssl x509 -noout -fingerprint -sha1 -in /opt/fog/snapins/ssl/CA/.fogCA.pem
    openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ca.cert.pem

  • @Sebastian-Roth
    First command:

    e15712aaee9359a90e94b46905018252  /opt/fog/snapins/ssl/CA/.fogCA.pem
    e15712aaee9359a90e94b46905018252  /var/www/html/fog/management/other/ca.cert.pem

    Second command:

    (stdin)= e1dc65877f9a55eb8c01744cf987bb50

    Third command:

    (stdin)= e1dc65877f9a55eb8c01744cf987bb50

    Fourth command:

    /var/www/fog/management/other/ssl/srvpublic.crt: OK

  • @Sebastian-Roth Ok, I’m gonna try that afterwards, kinda in the middle of something.

  • Moderator

    @Silv4n We need to make sure the CA cert file is the same in both placed on your FOG server:

    md5sum /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/html/fog/management/other/ca.cert.pem

    As well let’s check if the webserver certificate and key match:

    openssl x509 -noout -modulus -in /var/www/fog/management/other/ssl/srvpublic.crt | openssl md5
    openssl rsa -noout -modulus -in /opt/fog/snapins/ssl/.srvprivate.key | openssl md5

    And last but not least verify the webserver cert is being issued by the CA correctly:

    openssl verify -verbose -CAfile /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/fog/management/other/ssl/srvpublic.crt

    Post output of all the commands here.

  • @Sebastian-Roth No worries, there are no clients on the storage node, only on the master, but thanks for the info.

    Error Logs:

  • Moderator

    @Silv4n Please grab the install log files from fogproject/bin/error_logs/ of both nodes upload to a fileshare and post a link here.

    Be careful when using the --recreate-ca option when running the installer. As soon as you use fog-client software this will break communication with all your clients as they are bound to the CA cert.