• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

PXE Boot not working properly from Storage Node after Upgrade to 1.5.8

Scheduled Pinned Locked Moved Solved
FOG Problems
2
18
1.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    Silv4n
    last edited by Feb 21, 2020, 1:26 PM

    Hey guys,

    This is an addition to the request https://forums.fogproject.org/topic/14230/pxe-boot-not-working-properly-from-storage-node, which was solved with 1.5.7.

    I thought, that it would be good to just upgrade to 1.5.8, so that these things don’t happen again. However after upgrading both the master and storage, the PXE Boot fails yet again with the same error as yesterday. I’ve already tried --recreate-ca and --recreate-keys on the Storage Node.

    Error & certstat:
    https://imgur.com/a/OnJX221

    Thanks for any help

    1 Reply Last reply Reply Quote 0
    • S
      Sebastian Roth Moderator
      last edited by Feb 21, 2020, 4:27 PM

      @Silv4n Please grab the install log files from fogproject/bin/error_logs/ of both nodes upload to a fileshare and post a link here.

      Be careful when using the --recreate-ca option when running the installer. As soon as you use fog-client software this will break communication with all your clients as they are bound to the CA cert.

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      1 Reply Last reply Reply Quote 0
      • S
        Silv4n
        last edited by Silv4n Feb 21, 2020, 11:59 AM Feb 21, 2020, 5:59 PM

        @Sebastian-Roth No worries, there are no clients on the storage node, only on the master, but thanks for the info.

        Error Logs: https://bit.ly/2vSpoBy

        1 Reply Last reply Reply Quote 0
        • S
          Sebastian Roth Moderator
          last edited by Sebastian Roth Feb 21, 2020, 12:40 PM Feb 21, 2020, 6:40 PM

          @Silv4n We need to make sure the CA cert file is the same in both placed on your FOG server:

          md5sum /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/html/fog/management/other/ca.cert.pem
          

          As well let’s check if the webserver certificate and key match:

          openssl x509 -noout -modulus -in /var/www/fog/management/other/ssl/srvpublic.crt | openssl md5
          openssl rsa -noout -modulus -in /opt/fog/snapins/ssl/.srvprivate.key | openssl md5
          

          And last but not least verify the webserver cert is being issued by the CA correctly:

          openssl verify -verbose -CAfile /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/fog/management/other/ssl/srvpublic.crt
          

          Post output of all the commands here.

          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

          S 2 Replies Last reply Feb 21, 2020, 8:32 PM Reply Quote 0
          • S
            Silv4n @Sebastian Roth
            last edited by Feb 21, 2020, 8:32 PM

            @Sebastian-Roth Ok, I’m gonna try that afterwards, kinda in the middle of something.

            1 Reply Last reply Reply Quote 0
            • S
              Silv4n @Sebastian Roth
              last edited by Sebastian Roth Feb 22, 2020, 2:31 AM Feb 21, 2020, 9:40 PM

              @Sebastian-Roth
              First command:

              e15712aaee9359a90e94b46905018252  /opt/fog/snapins/ssl/CA/.fogCA.pem
              e15712aaee9359a90e94b46905018252  /var/www/html/fog/management/other/ca.cert.pem
              

              Second command:

              (stdin)= e1dc65877f9a55eb8c01744cf987bb50
              

              Third command:

              (stdin)= e1dc65877f9a55eb8c01744cf987bb50
              

              Fourth command:

              /var/www/fog/management/other/ssl/srvpublic.crt: OK
              
              1 Reply Last reply Reply Quote 0
              • S
                Sebastian Roth Moderator
                last edited by Feb 22, 2020, 8:48 AM

                @Silv4n All looks fine up to here. Can you please run ls -al /tftpboot/ and post output here. From the log output we see the installer ran at Feb 21 around 11 o’clock. Just want to see if the iPXE binaries have the same timestamp.

                As well I forgot one openssl command you shall run to see if the SHA1 fingerprint of the CA cert on the server matches the one we see in the picture:

                openssl x509 -noout -fingerprint -sha1 -in /opt/fog/snapins/ssl/CA/.fogCA.pem
                openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ca.cert.pem
                

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                S 1 Reply Last reply Feb 22, 2020, 9:09 AM Reply Quote 0
                • S
                  Silv4n @Sebastian Roth
                  last edited by Feb 22, 2020, 9:09 AM

                  @Sebastian-Roth
                  Command 1:

                  drwxr-xr-x  6 fogproject root    4096 Feb 20 11:34 .
                  drwxr-xr-x 26 root       root    4096 Feb 21 11:01 ..
                  drwxr-xr-x  4 fogproject root    4096 Feb 20 11:34 10secdelay
                  drwxr-xr-x  2 fogproject root    4096 Feb 21 08:23 arm64-efi
                  -rw-r-xr-x  1 fogproject root     868 Feb 21 11:02 boot.txt
                  -rw-r-xr-x  1 fogproject root     457 Feb 21 11:02 default.ipxe
                  drwxr-xr-x  2 fogproject root    4096 Feb 20 11:34 i386-efi
                  -rw-r-xr-x  1 fogproject root  227424 Feb 21 11:02 intel.efi
                  -rw-r-xr-x  1 fogproject root   99123 Feb 21 11:02 intel.kkpxe
                  -rw-r-xr-x  1 fogproject root   99171 Feb 21 11:02 intel.kpxe
                  -rw-r-xr-x  1 fogproject root   99146 Feb 21 11:02 intel.pxe
                  -rw-r-xr-x  1 fogproject root 1007360 Feb 21 11:02 ipxe.efi
                  -rw-r-xr-x  1 fogproject root  876544 Feb 21 11:02 ipxe.iso
                  -rw-r-xr-x  1 fogproject root  358066 Feb 21 11:02 ipxe.kkpxe
                  -rw-r-xr-x  1 fogproject root  358114 Feb 21 11:02 ipxe.kpxe
                  -rw-r-xr-x  1 fogproject root  357700 Feb 21 11:02 ipxe.krn
                  -rw-r-xr-x  1 fogproject root  357700 Feb 21 11:02 ipxe.lkrn
                  -rw-r-xr-x  1 fogproject root  358328 Feb 21 11:02 ipxe.pxe
                  -rw-r-xr-x  1 fogproject root 1409024 Feb 21 11:02 ipxe.usb
                  -rw-r-xr-x  1 fogproject root  123448 Feb 20 13:24 ldlinux.c32
                  -rw-r-xr-x  1 fogproject root  187820 Feb 20 13:24 libcom32.c32
                  -rw-r-xr-x  1 fogproject root   26468 Feb 20 13:24 libutil.c32
                  -rw-r-xr-x  1 fogproject root   26140 Feb 21 11:02 memdisk
                  -rw-r-xr-x  1 fogproject root   29208 Feb 20 13:24 menu.c32
                  -rw-r-xr-x  1 fogproject root  252768 Feb 21 11:02 ncm--ecm--axge.efi
                  -rw-r-xr-x  1 fogproject root   43210 Feb 20 13:24 pxelinux.0.old
                  drwxr-xr-x  2 fogproject root    4096 Feb 18 08:17 pxelinux.cfg
                  -rw-r-xr-x  1 fogproject root  226272 Feb 21 11:02 realtek.efi
                  -rw-r-xr-x  1 fogproject root   99950 Feb 21 11:02 realtek.kkpxe
                  -rw-r-xr-x  1 fogproject root   99998 Feb 21 11:02 realtek.kpxe
                  -rw-r-xr-x  1 fogproject root   99968 Feb 21 11:02 realtek.pxe
                  -rw-r-xr-x  1 fogproject root  225696 Feb 21 11:02 snp.efi
                  -rw-r-xr-x  1 fogproject root  225952 Feb 21 11:02 snponly.efi
                  -rw-r-xr-x  1 fogproject root   98645 Feb 21 11:02 undionly.kkpxe
                  -rw-r-xr-x  1 fogproject root   98693 Feb 21 11:02 undionly.kpxe
                  -rw-r-xr-x  1 fogproject root   98696 Feb 21 11:02 undionly.pxe
                  -rw-r-xr-x  1 fogproject root   29728 Feb 20 13:24 vesamenu.c32
                  

                  Command 2:

                  SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6F
                  

                  Command 3:

                  SHA1 Fingerprint=52:79:6A:2A:DB:DB:B2:97:93:0E:81:45:84:1B:92:D8:BB:6D:2B:6F
                  
                  1 Reply Last reply Reply Quote 0
                  • S
                    Sebastian Roth Moderator
                    last edited by Feb 22, 2020, 9:14 AM

                    @Silv4n Still all good. We shall find it soon I am sure. Try these commands:

                    openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt
                    echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | head

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    S 1 Reply Last reply Feb 22, 2020, 9:19 AM Reply Quote 0
                    • S
                      Sebastian Roth Moderator
                      last edited by Feb 22, 2020, 9:19 AM

                      @Silv4n And here is one more command:

                      echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | openssl x509  -noout -fingerprint
                      

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      1 Reply Last reply Reply Quote 0
                      • S
                        Silv4n @Sebastian Roth
                        last edited by Silv4n Feb 22, 2020, 3:20 AM Feb 22, 2020, 9:19 AM

                        @Sebastian-Roth

                        fogadmin@v-fogsrv02:~$ openssl x509 -noout -fingerprint -sha1 -in /var/www/html/fog/management/other/ssl/srvpublic.crt
                        SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93
                        fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | head
                        depth=1 CN = FOG Server CA
                        verify return:1
                        depth=0 CN = 10.144.1.22
                        verify return:1
                        DONE
                        CONNECTED(00000005)
                        ---
                        Certificate chain
                         0 s:CN = 10.144.1.22
                           i:CN = FOG Server CA
                         1 s:CN = FOG Server CA
                           i:CN = FOG Server CA
                        ---
                        Server certificate
                        -----BEGIN CERTIFICATE-----
                        
                        fogadmin@v-fogsrv02:~$ echo -n | openssl s_client -CAfile /var/www/html/fog/management/other/ca.cert.pem -connect 10.144.1.22:443 | openssl x509  -noout -fingerprint
                        depth=1 CN = FOG Server CA
                        verify return:1
                        depth=0 CN = 10.144.1.22
                        verify return:1
                        DONE
                        SHA1 Fingerprint=83:7B:9D:57:E9:11:51:83:46:20:7F:81:04:A2:23:44:A7:68:34:93
                        
                        1 Reply Last reply Reply Quote 0
                        • S
                          Sebastian Roth Moderator
                          last edited by Feb 22, 2020, 9:23 AM

                          @Silv4n This is really strange. All the certificates seem perfectly fine and match the fingerprints we see in the picture you posted initially. I just did a fresh clean install here and it worked out of the box. Though this is a master server only. Let me try adding a storage node and see if that makes a difference.

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          S 1 Reply Last reply Feb 22, 2020, 9:24 AM Reply Quote 0
                          • S
                            Silv4n @Sebastian Roth
                            last edited by Feb 22, 2020, 9:24 AM

                            @Sebastian-Roth That’s the storage node

                            1 Reply Last reply Reply Quote 1
                            • S
                              Sebastian Roth Moderator
                              last edited by Feb 22, 2020, 9:25 AM

                              @Silv4n Let’s switch over to chat (chat bubble in the top right corner).

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              S 1 Reply Last reply Feb 22, 2020, 1:09 PM Reply Quote 0
                              • S
                                Silv4n @Sebastian Roth
                                last edited by Feb 22, 2020, 1:09 PM

                                @Sebastian-Roth unfortunatly the can’t display all of it: https://imgur.com/a/OKdQzwh

                                1 Reply Last reply Reply Quote 0
                                • S
                                  Sebastian Roth Moderator
                                  last edited by Sebastian Roth Feb 22, 2020, 9:33 AM Feb 22, 2020, 3:07 PM

                                  @Silv4n Ok, unfortunately not of much help yet. Please recompile but leave out the tls, in DEBUG parameter…

                                  make EMBED=ipxescript DEBUG=x509,validator bin/undionly.kpxe CERT=/opt/fog/snapins/ssl/CA/.fogCA.pem TRUST=/opt/fog/snapins/ssl/CA/.fogCA.pem
                                  cp bin/undionly.kpxe /tftpboot
                                  

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  S 1 Reply Last reply Feb 22, 2020, 3:40 PM Reply Quote 0
                                  • S
                                    Silv4n @Sebastian Roth
                                    last edited by Feb 22, 2020, 3:40 PM

                                    @Sebastian-Roth https://imgur.com/a/j0WJInw

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      Sebastian Roth Moderator
                                      last edited by Feb 25, 2020, 7:24 PM

                                      After some extended research I figured out this was caused by the build script not re-generating the trusted root part of the code compiled into the iPXE binaries. It’s really easy to fix and I pushed a fix to both dev-branch and working-1.6 so we hopefully never run into this again.

                                      cd path/to/fogproject/bin/
                                      touch ../../ipxe/src/crypto/rootcert.c
                                      rm /tftpboot/undionly.kkpxe
                                      ./installfog.sh
                                      

                                      On that way I learned a couple of things about iPXE booting over HTTPS and so I hope we can find most upcoming issues more quickly from now on.

                                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                      1 Reply Last reply Reply Quote 0
                                      • 1 / 1
                                      1 / 1
                                      • First post
                                        1/18
                                        Last post

                                      185

                                      Online

                                      12.0k

                                      Users

                                      17.3k

                                      Topics

                                      155.2k

                                      Posts
                                      Copyright © 2012-2024 FOG Project