• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

ipxe boot slow after changing to HTTPS

Scheduled Pinned Locked Moved Unsolved
FOG Problems
3
38
2.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • D
    DBCountMan
    last edited by Feb 13, 2023, 9:01 PM

    Running v1.5.9.231. I had to enable HTTPS since we use LDAP to authenticate when logging into FOG via web UI and the ipxe menu. Since then, anything pulled from either HTTP/HTTPS takes significantly more time to load. TFTP transfers are still fast and unaffected. The only transfers from HTTP that seem to be fine are those final WIM downloads (from the ipxe menu), but the files before that all take about 5 seconds to start fetching. It seems more like a delay than actual transfer speed.

    Is this normal behavior for HTTPS?

    J 1 Reply Last reply Feb 25, 2023, 7:57 PM Reply Quote 0
    • S
      Sebastian Roth Moderator
      last edited by Feb 15, 2023, 5:28 AM

      @brakcounty No I don’t think the described behavior is “normal”. Please be more specific on what files take long. Is it bzImage and init.xz?

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      D 1 Reply Last reply Feb 15, 2023, 7:46 PM Reply Quote 0
      • D
        DBCountMan @Sebastian Roth
        last edited by DBCountMan Feb 15, 2023, 3:10 PM Feb 15, 2023, 7:46 PM

        @Sebastian-Roth Once I start the PXE boot process, the boot kernel loads quick over TFTP, then the next part that loads boot.php takes longer. After I hit ESC to load the authentication screen, I enter my creds, then the next part where it loads the boot.php twice, then bg.png. After I get to the actual FOG ipxe menu, then load a custom ipxe entry such as WinPE, anything that loads via HTTP has a delay before starting the download. For my example of WinPE, the BCD, boot.sdi, bootmgr.efi, then finally the wim file which downloads normally, but there is a delay now before the download actually starts.

        Does this make sense? My description of the issue I mean.

        1 Reply Last reply Reply Quote 0
        • D
          DBCountMan
          last edited by Feb 15, 2023, 8:17 PM

          @Sebastian-Roth Update: Yes the bzImage and init.xz also take longer to load now. Assuming because they are served via HTTP?

          1 Reply Last reply Reply Quote 0
          • S
            Sebastian Roth Moderator
            last edited by Feb 16, 2023, 7:13 AM

            @brakcounty As I said this is not normal I reckon. From what we heard so far I can imagine some kind of security gatway causing the delay while checking HTTPS traffic. Though on the other hand that would need to be a highly sofisticated transparent proxy setup. Please check with your network team.

            Other than that I suggest you do manual tests in your browser. Open Chrome or Firefox development tool bar and then load the boot.php and other URLs you find having the delay. See if you can find out what is causing it.

            As a next step you probably need to install wireshark on the PC and capture the network traffic. Feel free to send me a private message with a download link to the saved wireshark pcap file if you need help with reading it.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            D 1 Reply Last reply Feb 16, 2023, 4:05 PM Reply Quote 1
            • D
              DBCountMan @Sebastian Roth
              last edited by Feb 16, 2023, 4:05 PM

              @Sebastian-Roth That was my next guess but wanted to confirm that it wasn’t normal. Thanks!

              1 Reply Last reply Reply Quote 0
              • D
                DBCountMan
                last edited by DBCountMan Feb 16, 2023, 10:17 AM Feb 16, 2023, 4:12 PM

                Ok so I browsed these links on my desktop using Firefox, and they loaded instantly:
                https://<fog-ip>/fog/service/ipxe/boot.php - 96ms
                https://<fog-ip>/fog/service/ipxe/boot.php##params - 96ms

                We do in fact scan all traffic, but I noticed that the delay is only during ipxe ops.

                I’ll run wireshark next.

                1 Reply Last reply Reply Quote 0
                • S
                  Sebastian Roth Moderator
                  last edited by Feb 16, 2023, 9:24 PM

                  @brakcounty said in ipxe boot slow after changing to HTTPS:

                  We do in fact scan all traffic, but I noticed that the delay is only during ipxe ops.

                  So maybe the scan only happens (or is only being delayed) when the request header is not a normal browser?

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  D 1 Reply Last reply Feb 17, 2023, 7:42 PM Reply Quote 0
                  • D
                    DBCountMan @Sebastian Roth
                    last edited by Feb 17, 2023, 7:42 PM

                    @Sebastian-Roth I confirmed with our network team that we scan traffic by protocol, so even if an app that isn’t a browser makes an HTTP/S connection, it will get scanned.

                    1 Reply Last reply Reply Quote 0
                    • S
                      Sebastian Roth Moderator
                      last edited by Feb 18, 2023, 7:15 AM

                      @brakcounty So we still need to figure out why the browser gets a quick response while iPXE does not?! Probably using tcpdump/wireshark as mentioned or even better asking your network team to look into it.

                      You can to more tests as well, either download wget or curl for windows to test. Or you can boot up a Linux live OS CD to do the same testing.

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      D 1 Reply Last reply Feb 21, 2023, 6:34 PM Reply Quote 0
                      • D
                        DBCountMan @Sebastian Roth
                        last edited by DBCountMan Feb 21, 2023, 1:40 PM Feb 21, 2023, 6:34 PM

                        @Sebastian-Roth I just did some testing on my desktop, which is on a different vlan than the fog server, but shouldn’t matter:
                        On my Windows console:
                        curl and wget https://10.240.160.59/fog/service/ipxe/boot.php both show this message:
                        The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

                        I also have Linux sub-system on my Windows box:
                        curl https://10.240.160.59/fog/service/ipxe/boot.php
                        curl: (60) SSL certificate problem: self signed certificate in certificate chain

                        wget https://10.240.160.59/fog/service/ipxe/boot.php
                        –2023-02-21 13:26:44-- https://10.240.160.59/fog/service/ipxe/boot.php
                        Connecting to 10.240.160.59:443… connected.
                        ERROR: cannot verify 10.240.160.59’s certificate, issued by ‘CN=FOG Server CA’:
                        Self-signed certificate encountered.
                        To connect to 10.240.160.59 insecurely, use `–no-check-certificate’.

                        Obviously, I don’t have FOGs cert installed on my Windows PC, which I don’t need since I’m not doing any pxe ops from it.

                        I’m going to see if I can set up a VM to pxe boot while running wireshark in the bg.

                        EDIT: I have a Hyper-V vm booting to FOG via USB Boot method. I have a vm storage volume that has the bootx64.efi and I boot from that to start the iPXE boot process.

                        1 Reply Last reply Reply Quote 0
                        • S
                          Sebastian Roth Moderator
                          last edited by Feb 21, 2023, 10:42 PM

                          @brakcounty Sure thing you need to tell curl/wget to ignore/accept the non-official certificate: wget --no-check-certificate ... or curl -k ...

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          D 1 Reply Last reply Feb 22, 2023, 6:16 PM Reply Quote 0
                          • D
                            DBCountMan @Sebastian Roth
                            last edited by Feb 22, 2023, 6:16 PM

                            @Sebastian-Roth Ran from a console, instant. Still working on getting an accurate pcap.

                            root@mypc:~/scripts# curl https://fogserverip/fog/service/ipxe/boot.php -k
                            #!ipxe
                            set fog-ip fogserverip
                            set fog-webroot fog
                            set boot-url https://${fog-ip}/${fog-webroot}
                            set storage-ip fogserverip
                            set keymap us
                            cpuid --ext 29 && set arch x86_64 || set arch i386
                            iseq ${platform} efi && set key 0x1b || set key 0x1b
                            iseq ${platform} efi && set keyName ESC || set keyName Escape
                            prompt --key ${key} --timeout 3000 Booting... (Press ${keyName} to access the menu) && goto menuAccess || exit
                            :menuAccess
                            login
                            params
                            param mac0 ${net0/mac}
                            param arch ${arch}
                            param platform ${platform}
                            param username ${username}
                            param password ${password}
                            param menuaccess 1
                            param debug 1
                            param sysuuid ${uuid}
                            isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
                            isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
                            :bootme
                            chain -ar https://fogserverip/fog/service/ipxe/boot.php##params
                            root@mypc:~/scripts# wget https://fogserverip/fog/service/ipxe/boot.php --no-check-certificate
                            --2023-02-22 11:54:54--  https://fogserverip/fog/service/ipxe/boot.php
                            Connecting to fogserverip:443... connected.
                            WARNING: cannot verify fogserverip's certificate, issued by ‘CN=FOG Server CA’:
                             Self-signed certificate encountered.
                            HTTP request sent, awaiting response... 200 OK
                            Length: unspecified [text/plain]
                            Saving to: ‘boot.php.1’
                            
                            boot.php.1                                               [ <=>                                                                                                                ]     813  --.-KB/s    in 0s
                            
                            2023-02-22 11:54:55 (180 MB/s) - ‘boot.php.1’ saved [813]
                            
                            1 Reply Last reply Reply Quote 0
                            • S
                              Sebastian Roth Moderator
                              last edited by Feb 22, 2023, 9:01 PM

                              @brakcounty Ok, from the tests we have done so far it kind of looks like this is going to be an iPXE network driver issue. Interesting I have not thought of this before.

                              Have you tried different iPXE binaries yet? ipxe.efi vs. snponly.efi? ipxe.pxe vs. undionly.kkpxe?

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              D 1 Reply Last reply Feb 23, 2023, 2:24 PM Reply Quote 0
                              • D
                                DBCountMan @Sebastian Roth
                                last edited by Feb 23, 2023, 2:24 PM

                                @Sebastian-Roth I haven’t tried different binaries yet. Wouldn’t I have to recompile them to use HTTPS? Did the -s switch during setup automatically compile all those efi binaries and place them into /tftproot?

                                1 Reply Last reply Reply Quote 0
                                • D
                                  DBCountMan
                                  last edited by Feb 23, 2023, 4:47 PM

                                  Now that you’ve mentioned ipxe driver issue, it seems more likely. The delay is longer on my xencenter VMs vs VirtualBox VMs and physical PCs.

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    Sebastian Roth Moderator
                                    last edited by Feb 23, 2023, 6:07 PM

                                    @brakcounty said in ipxe boot slow after changing to HTTPS:

                                    Did the -s switch during setup automatically compile all those efi binaries and place them into /tftproot?

                                    Yes.

                                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                    1 Reply Last reply Reply Quote 0
                                    • D
                                      DBCountMan
                                      last edited by Feb 24, 2023, 7:37 PM

                                      Booting from snponly.efi doesn’t recognize the network adapter. I tried using Intel and ParaVirt in VirtualBox.
                                      ff997b7c-8450-4744-bdb4-a47975d77462-image.png

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        Sebastian Roth Moderator
                                        last edited by Feb 25, 2023, 1:46 PM

                                        @brakcounty Try out different ones, like intel.efi for example.

                                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                        D 1 Reply Last reply Mar 6, 2023, 6:50 PM Reply Quote 0
                                        • J
                                          JJ Fullmer Testers @DBCountMan
                                          last edited by Feb 25, 2023, 7:57 PM

                                          @brakcounty and @Sebastian-Roth
                                          I recently did a fresh install of a fog dev server and did https and experienced similar slowness on the kernel loading.
                                          I’ll give some of this testing a try and report back to see if this is maybe more common than we think.

                                          Have you tried the FogApi powershell module? It's pretty cool IMHO
                                          https://github.com/darksidemilk/FogApi
                                          https://fogapi.readthedocs.io/en/latest/
                                          https://www.powershellgallery.com/packages/FogApi
                                          https://forums.fogproject.org/topic/12026/powershell-api-module

                                          1 Reply Last reply Reply Quote 0
                                          • 1
                                          • 2
                                          • 1 / 2
                                          1 / 2
                                          • First post
                                            2/38
                                            Last post

                                          149

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project