• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

ipxe boot slow after changing to HTTPS

Scheduled Pinned Locked Moved Unsolved
FOG Problems
3
38
2.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • D
    DBCountMan @Sebastian Roth
    last edited by Mar 6, 2023, 6:50 PM

    @Sebastian-Roth I tried intel.efi, still slow.

    1 Reply Last reply Reply Quote 0
    • S
      Sebastian Roth Moderator
      last edited by Mar 6, 2023, 10:15 PM

      @brakcounty said in ipxe boot slow after changing to HTTPS:

      The delay is longer on my xencenter VMs vs VirtualBox VMs and physical PCs.

      Let’s go back to this information. Are physical PCs as fast as it used to be with plain HTTP?

      I do use VirtualBox in my test setups and never saw it going slow on HTTPS.

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      D 1 Reply Last reply Mar 7, 2023, 1:15 PM Reply Quote 0
      • D
        DBCountMan @Sebastian Roth
        last edited by Mar 7, 2023, 1:15 PM

        @Sebastian-Roth physical PCs are still slower on HTTPS than HTTP. I was saying that the delay is exacerbated on VMs, especially slow (the slowest in fact) on XCP-NG guests. VirtualBox is better, physical is fastest. All three environments are still slower using HTTPS vs HTTP. I remember how instant HTTP was on any platform.

        1 Reply Last reply Reply Quote 0
        • D
          DBCountMan
          last edited by Mar 7, 2023, 3:14 PM

          I just want to reiterate that when I say slow/fast, I’m referring to the time it takes to initiate a download (get) of a file via HTTPS. Once the download starts, then the speed is fine.

          1 Reply Last reply Reply Quote 0
          • S
            Sebastian Roth Moderator
            last edited by Sebastian Roth Mar 7, 2023, 11:06 PM Mar 8, 2023, 5:04 AM

            @brakcounty said:

            I was saying that the delay is exacerbated on VMs, especially slow (the slowest in fact) on XCP-NG guests. VirtualBox is better, physical is fastest.

            Although I am not sure this is important I would say we better keep that information afloat in the back of our minds.

            Ran from a console, instant. Still working on getting an accurate pcap.

            Ok, we need to get back to that point then.

            1. Please schedule a debug (capture or deploy) task for any machine you see this issue on. Start it up and hit ENTER twice to get to the shell.
              Then run wget --no-check-certificate https://fogserverip/fog/service/ipxe/bzImage and let us know if this is starting instantly or delayed.
            2. In the FOG web UI go to FOG Configuration -> iPXE New Menu Entry and enter the following information:
              Menu Item: fog.ipxeshell
              Description: iPXE shell
              Parameters: shell || goto MENU
              Boot Options: leave empty
              Default Item: unchecked
              Hot Key Enabled: unchecked
              Hot Key to use: leave empty
              Menu Show with: Registered Hosts
              Now boot up a machine/VM having the issue, select the iPXE shell and run command kernel bzImage and once again let us know if this is starting instantly or delayed.

            Outcomes:

            • If both those show the delay symptom we are surely talking about a very crude network issue that is only seen in FOS/iPXE but not when the OS is booted - very unlikely. But if that’s the case you need to look into packet capturing as suggested before!!
            • If the first test is instant but the second one is delayed we seem to have an iPXE issue - on the one hand I have never seen this on my HTTPS setups but also this is the most likely outcome from my perspective.
            • If the first one is delayed but the second one gets an instant response - kind of impossible - then I have no idea and we need to re-think the whole case.
            • And finally, if both tests yield in an instant response I would be puzzled as well. Then we’d need to dig into the differences between manual test and the normal PXE booting sequence.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            1 Reply Last reply Reply Quote 0
            • D
              DBCountMan
              last edited by Mar 8, 2023, 3:28 PM

              @Sebastian-Roth I pm’d you a pcap

              Ran these tests on my hyper-v and xcp vms:

              1. In the FOG debug console (Both Hyper-V and XCP showed this result)
              wget --no-check-certificate https://fogserverip/fog/service/ipxe/bzImage
              wget: not an http or ftp url: https://fogserverip/fog/service/ipxe/bzImage
              
              1. kernel bzImage took about 3-4 seconds on hyper-v, 10 seconds on xcp, then returned with
              bzImage...ok
              
              1 Reply Last reply Reply Quote 0
              • S
                Sebastian Roth Moderator
                last edited by Mar 8, 2023, 8:26 PM

                @brakcounty said in ipxe boot slow after changing to HTTPS:

                wget: not an http or ftp url: https://fogserverip/fog/service/ipxe/bzImage

                I have to admit that I have not tried it myself yet but I’d be pretty amazed if the wget binary we ship is not able to handle the HTTPS protocol. Anyhow, can you try curl -v -k https://fogserverip/fog/service/ipxe/bzImage instead?

                kernel bzImage took about 3-4 seconds on hyper-v, 10 seconds on xcp, then returned with

                Is this slower or faster than you see when PXE booting into a task?

                I pm’d you a pcap

                The first TCP SYN send by the client to open the connection should be answered by a SYN,ACK by the server but in the PCAP we see a simple ACK which wireshark tells us is “ACKed unseen segment” - like a packet from a different connection (but on the same ports!). This is very unusual! Then the client re-sends the initial SYN packet and gets a proper SYN,ACK back, returns an ACK to properly finish the TCP three way handshake.

                Beside this strange behavior I wonder where the delay would happen. The first 9-10 seconds take for the DHCP DORA. The TCP handshake starts at 9.88 and goes straight into the SSL session setup. Between “Server Key Exchange, Server Hello Done” and “Client Key Exchange” there is a 2.5 second delay (caused by the client waiting) which I don’t find normal. Though I can imagine this is due to crypto algorithm calculations. The rest of the TCP communication looks to be fast.

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                D 1 Reply Last reply Mar 10, 2023, 7:28 PM Reply Quote 0
                • D
                  DBCountMan
                  last edited by DBCountMan Mar 10, 2023, 1:27 PM Mar 10, 2023, 7:26 PM

                  Ran the curl command, instant.
                  2cd4f3bb-89f4-40f9-9c98-50e1c4451d95-image.png

                  1 Reply Last reply Reply Quote 0
                  • D
                    DBCountMan @Sebastian Roth
                    last edited by DBCountMan Mar 10, 2023, 1:28 PM Mar 10, 2023, 7:28 PM

                    @Sebastian-Roth said in ipxe boot slow after changing to HTTPS:

                    • like a packet from a different connection (but on the same ports!)

                    This could be the NAT’d VM IP. I ran wireshark on the Default Hyper-V Switch adapter.

                    1 Reply Last reply Reply Quote 0
                    • S
                      Sebastian Roth Moderator
                      last edited by Mar 10, 2023, 9:46 PM

                      @Sebastian-Roth said in ipxe boot slow after changing to HTTPS:

                      If the first test is instant but the second one is delayed we seem to have an iPXE issue - on the one hand I have never seen this on my HTTPS setups but also this is the most likely outcome from my perspective.

                      So this is what we are at right now, right?? And you tested this on different machines, VMs as well as hardware.

                      I will try to replicate the issue. If I can’t we should schedule for a debug session together some time next week.

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      D 1 Reply Last reply Mar 13, 2023, 2:40 PM Reply Quote 0
                      • D
                        DBCountMan @Sebastian Roth
                        last edited by Mar 13, 2023, 2:40 PM

                        @Sebastian-Roth Definitely looks like it is isolated to ipxe.

                        @Sebastian-Roth said in ipxe boot slow after changing to HTTPS:

                        I have never seen this on my HTTPS setups

                        Out of curiosity, what NICs do you typically run ipxe on?

                        1 Reply Last reply Reply Quote 0
                        • S
                          Sebastian Roth Moderator
                          last edited by Mar 14, 2023, 1:17 PM

                          @brakcounty Didn’t find the time to test on my side yet. Will do in the next days and let you know.

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          D 1 Reply Last reply Mar 15, 2023, 5:06 PM Reply Quote 1
                          • D
                            DBCountMan @Sebastian Roth
                            last edited by Mar 15, 2023, 5:06 PM

                            @Sebastian-Roth Cool thanks, much apppreciated. By the way, this isn’t an operation-breaking critical issue, so take your time.

                            1 Reply Last reply Reply Quote 0
                            • S
                              Sebastian Roth Moderator
                              last edited by Mar 20, 2023, 12:02 AM

                              @brakcounty I still have not found enough time to do a test setup…

                              Out of curiosity, what NICs do you typically run ipxe on?

                              The default on Linux virtualbox: Intel PRO/1000 MT Desktop (82540EM)

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              D 1 Reply Last reply Mar 20, 2023, 1:25 PM Reply Quote 0
                              • D
                                DBCountMan @Sebastian Roth
                                last edited by Mar 20, 2023, 1:25 PM

                                @Sebastian-Roth said in ipxe boot slow after changing to HTTPS:

                                The default on Linux virtualbox: Intel PRO/1000 MT Desktop (82540EM)

                                Hmm. Assuming you’re booting legacy pxe instead of UEFI, since UEFI PXE boot on Vbox requires the Paravirtualized Network Adapter (virtio) adapter. Not that it made make a difference for me at least once ipxe is loaded.

                                1 Reply Last reply Reply Quote 0
                                • S
                                  Sebastian Roth Moderator
                                  last edited by Mar 21, 2023, 1:40 PM

                                  @brakcounty said in ipxe boot slow after changing to HTTPS:

                                  Assuming you’re booting legacy pxe instead of UEFI, since UEFI PXE boot on Vbox requires the Paravirtualized Network Adapter (virtio) adapter

                                  Yes, legacy. Because I never got UEFI PXE booting to work in vbox on Linux.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  D 1 Reply Last reply Mar 21, 2023, 3:27 PM Reply Quote 0
                                  • D
                                    DBCountMan @Sebastian Roth
                                    last edited by Mar 21, 2023, 3:27 PM

                                    @Sebastian-Roth said in ipxe boot slow after changing to HTTPS:

                                    never got UEFI PXE booting to work in vbox on Linux

                                    Even when using the Paravirtualized Network Adapter in VBox?

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      Sebastian Roth Moderator
                                      last edited by Mar 25, 2023, 11:29 AM

                                      @brakcounty said in ipxe boot slow after changing to HTTPS:

                                      Even when using the Paravirtualized Network Adapter in VBox?

                                      Learning something new every day! Wow, didn’t know this and never tried it before. Thanks!!!

                                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                      1 Reply Last reply Reply Quote 0
                                      • 1
                                      • 2
                                      • 2 / 2
                                      2 / 2
                                      • First post
                                        28/38
                                        Last post

                                      207

                                      Online

                                      12.0k

                                      Users

                                      17.3k

                                      Topics

                                      155.2k

                                      Posts
                                      Copyright © 2012-2024 FOG Project