ipxe boot slow after changing to HTTPS
-
@Sebastian-Roth I pm’d you a pcap
Ran these tests on my hyper-v and xcp vms:
- In the FOG debug console (Both Hyper-V and XCP showed this result)
wget --no-check-certificate https://fogserverip/fog/service/ipxe/bzImage wget: not an http or ftp url: https://fogserverip/fog/service/ipxe/bzImage- kernel bzImage took about 3-4 seconds on hyper-v, 10 seconds on xcp, then returned with
bzImage...ok -
@brakcounty said in ipxe boot slow after changing to HTTPS:
wget: not an http or ftp url: https://fogserverip/fog/service/ipxe/bzImage
I have to admit that I have not tried it myself yet but I’d be pretty amazed if the wget binary we ship is not able to handle the HTTPS protocol. Anyhow, can you try
curl -v -k https://fogserverip/fog/service/ipxe/bzImageinstead?kernel bzImage took about 3-4 seconds on hyper-v, 10 seconds on xcp, then returned with
Is this slower or faster than you see when PXE booting into a task?
I pm’d you a pcap
The first TCP SYN send by the client to open the connection should be answered by a SYN,ACK by the server but in the PCAP we see a simple ACK which wireshark tells us is “ACKed unseen segment” - like a packet from a different connection (but on the same ports!). This is very unusual! Then the client re-sends the initial SYN packet and gets a proper SYN,ACK back, returns an ACK to properly finish the TCP three way handshake.
Beside this strange behavior I wonder where the delay would happen. The first 9-10 seconds take for the DHCP DORA. The TCP handshake starts at 9.88 and goes straight into the SSL session setup. Between “Server Key Exchange, Server Hello Done” and “Client Key Exchange” there is a 2.5 second delay (caused by the client waiting) which I don’t find normal. Though I can imagine this is due to crypto algorithm calculations. The rest of the TCP communication looks to be fast.
-
Ran the curl command, instant.
-
@Sebastian-Roth said in ipxe boot slow after changing to HTTPS:
- like a packet from a different connection (but on the same ports!)
This could be the NAT’d VM IP. I ran wireshark on the Default Hyper-V Switch adapter.
-
@Sebastian-Roth said in ipxe boot slow after changing to HTTPS:
If the first test is instant but the second one is delayed we seem to have an iPXE issue - on the one hand I have never seen this on my HTTPS setups but also this is the most likely outcome from my perspective.
So this is what we are at right now, right?? And you tested this on different machines, VMs as well as hardware.
I will try to replicate the issue. If I can’t we should schedule for a debug session together some time next week.
-
@Sebastian-Roth Definitely looks like it is isolated to ipxe.
@Sebastian-Roth said in ipxe boot slow after changing to HTTPS:
I have never seen this on my HTTPS setups
Out of curiosity, what NICs do you typically run ipxe on?
-
@brakcounty Didn’t find the time to test on my side yet. Will do in the next days and let you know.
-
@Sebastian-Roth Cool thanks, much apppreciated. By the way, this isn’t an operation-breaking critical issue, so take your time.
-
@brakcounty I still have not found enough time to do a test setup…
Out of curiosity, what NICs do you typically run ipxe on?
The default on Linux virtualbox: Intel PRO/1000 MT Desktop (82540EM)
-
@Sebastian-Roth said in ipxe boot slow after changing to HTTPS:
The default on Linux virtualbox: Intel PRO/1000 MT Desktop (82540EM)
Hmm. Assuming you’re booting legacy pxe instead of UEFI, since UEFI PXE boot on Vbox requires the Paravirtualized Network Adapter (virtio) adapter. Not that it made make a difference for me at least once ipxe is loaded.
-
@brakcounty said in ipxe boot slow after changing to HTTPS:
Assuming you’re booting legacy pxe instead of UEFI, since UEFI PXE boot on Vbox requires the Paravirtualized Network Adapter (virtio) adapter
Yes, legacy. Because I never got UEFI PXE booting to work in vbox on Linux.
-
@Sebastian-Roth said in ipxe boot slow after changing to HTTPS:
never got UEFI PXE booting to work in vbox on Linux
Even when using the Paravirtualized Network Adapter in VBox?
-
@brakcounty said in ipxe boot slow after changing to HTTPS:
Even when using the Paravirtualized Network Adapter in VBox?
Learning something new every day! Wow, didn’t know this and never tried it before. Thanks!!!