ipxe boot slow after changing to HTTPS

[Sebastian Roth]

@brakcounty said in ipxe boot slow after changing to HTTPS:

We do in fact scan all traffic, but I noticed that the delay is only during ipxe ops.

So maybe the scan only happens (or is only being delayed) when the request header is not a normal browser?

[DBCountMan]

@Sebastian-Roth I confirmed with our network team that we scan traffic by protocol, so even if an app that isn’t a browser makes an HTTP/S connection, it will get scanned.

[Sebastian Roth]

@brakcounty So we still need to figure out why the browser gets a quick response while iPXE does not?! Probably using tcpdump/wireshark as mentioned or even better asking your network team to look into it.

You can to more tests as well, either download wget or curl for windows to test. Or you can boot up a Linux live OS CD to do the same testing.

[DBCountMan]

@Sebastian-Roth I just did some testing on my desktop, which is on a different vlan than the fog server, but shouldn’t matter:
On my Windows console:
curl and wget https://10.240.160.59/fog/service/ipxe/boot.php both show this message:
The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

I also have Linux sub-system on my Windows box:
curl https://10.240.160.59/fog/service/ipxe/boot.php
curl: (60) SSL certificate problem: self signed certificate in certificate chain

wget https://10.240.160.59/fog/service/ipxe/boot.php
–2023-02-21 13:26:44-- https://10.240.160.59/fog/service/ipxe/boot.php
Connecting to 10.240.160.59:443… connected.
ERROR: cannot verify 10.240.160.59’s certificate, issued by ‘CN=FOG Server CA’:
Self-signed certificate encountered.
To connect to 10.240.160.59 insecurely, use `–no-check-certificate’.

Obviously, I don’t have FOGs cert installed on my Windows PC, which I don’t need since I’m not doing any pxe ops from it.

I’m going to see if I can set up a VM to pxe boot while running wireshark in the bg.

EDIT: I have a Hyper-V vm booting to FOG via USB Boot method. I have a vm storage volume that has the bootx64.efi and I boot from that to start the iPXE boot process.

[Sebastian Roth]

@brakcounty Sure thing you need to tell curl/wget to ignore/accept the non-official certificate: wget --no-check-certificate ... or curl -k ...

[DBCountMan]

@Sebastian-Roth Ran from a console, instant. Still working on getting an accurate pcap.

root@mypc:~/scripts# curl https://fogserverip/fog/service/ipxe/boot.php -k
#!ipxe
set fog-ip fogserverip
set fog-webroot fog
set boot-url https://${fog-ip}/${fog-webroot}
set storage-ip fogserverip
set keymap us
cpuid --ext 29 && set arch x86_64 || set arch i386
iseq ${platform} efi && set key 0x1b || set key 0x1b
iseq ${platform} efi && set keyName ESC || set keyName Escape
prompt --key ${key} --timeout 3000 Booting... (Press ${keyName} to access the menu) && goto menuAccess || exit
:menuAccess
login
params
param mac0 ${net0/mac}
param arch ${arch}
param platform ${platform}
param username ${username}
param password ${password}
param menuaccess 1
param debug 1
param sysuuid ${uuid}
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:bootme
chain -ar https://fogserverip/fog/service/ipxe/boot.php##params
root@mypc:~/scripts# wget https://fogserverip/fog/service/ipxe/boot.php --no-check-certificate
--2023-02-22 11:54:54--  https://fogserverip/fog/service/ipxe/boot.php
Connecting to fogserverip:443... connected.
WARNING: cannot verify fogserverip's certificate, issued by ‘CN=FOG Server CA’:
 Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘boot.php.1’

boot.php.1                                               [ <=>                                                                                                                ]     813  --.-KB/s    in 0s

2023-02-22 11:54:55 (180 MB/s) - ‘boot.php.1’ saved [813]

[Sebastian Roth]

@brakcounty Ok, from the tests we have done so far it kind of looks like this is going to be an iPXE network driver issue. Interesting I have not thought of this before.

Have you tried different iPXE binaries yet? ipxe.efi vs. snponly.efi? ipxe.pxe vs. undionly.kkpxe?

[DBCountMan]

@Sebastian-Roth I haven’t tried different binaries yet. Wouldn’t I have to recompile them to use HTTPS? Did the -s switch during setup automatically compile all those efi binaries and place them into /tftproot?

[DBCountMan]

Now that you’ve mentioned ipxe driver issue, it seems more likely. The delay is longer on my xencenter VMs vs VirtualBox VMs and physical PCs.

[Sebastian Roth]

@brakcounty said in ipxe boot slow after changing to HTTPS:

Did the -s switch during setup automatically compile all those efi binaries and place them into /tftproot?

Yes.

[DBCountMan]

Booting from snponly.efi doesn’t recognize the network adapter. I tried using Intel and ParaVirt in VirtualBox.

[Sebastian Roth]

@brakcounty Try out different ones, like intel.efi for example.

[JJ Fullmer]

@brakcounty and @Sebastian-Roth
I recently did a fresh install of a fog dev server and did https and experienced similar slowness on the kernel loading.
I’ll give some of this testing a try and report back to see if this is maybe more common than we think.

[DBCountMan]

@Sebastian-Roth I tried intel.efi, still slow.

[Sebastian Roth]

@brakcounty said in ipxe boot slow after changing to HTTPS:

The delay is longer on my xencenter VMs vs VirtualBox VMs and physical PCs.

Let’s go back to this information. Are physical PCs as fast as it used to be with plain HTTP?

I do use VirtualBox in my test setups and never saw it going slow on HTTPS.

[DBCountMan]

@Sebastian-Roth physical PCs are still slower on HTTPS than HTTP. I was saying that the delay is exacerbated on VMs, especially slow (the slowest in fact) on XCP-NG guests. VirtualBox is better, physical is fastest. All three environments are still slower using HTTPS vs HTTP. I remember how instant HTTP was on any platform.

[DBCountMan]

I just want to reiterate that when I say slow/fast, I’m referring to the time it takes to initiate a download (get) of a file via HTTPS. Once the download starts, then the speed is fine.

[Sebastian Roth]

@brakcounty said:

I was saying that the delay is exacerbated on VMs, especially slow (the slowest in fact) on XCP-NG guests. VirtualBox is better, physical is fastest.

Although I am not sure this is important I would say we better keep that information afloat in the back of our minds.

Ran from a console, instant. Still working on getting an accurate pcap.

Ok, we need to get back to that point then.

1. Please schedule a debug (capture or deploy) task for any machine you see this issue on. Start it up and hit ENTER twice to get to the shell.
   Then run wget --no-check-certificate https://fogserverip/fog/service/ipxe/bzImage and let us know if this is starting instantly or delayed.
2. In the FOG web UI go to FOG Configuration -> iPXE New Menu Entry and enter the following information:
   Menu Item: fog.ipxeshell
   Description: iPXE shell
   Parameters: shell || goto MENU
   Boot Options: leave empty
   Default Item: unchecked
   Hot Key Enabled: unchecked
   Hot Key to use: leave empty
   Menu Show with: Registered Hosts
   Now boot up a machine/VM having the issue, select the iPXE shell and run command kernel bzImage and once again let us know if this is starting instantly or delayed.

Outcomes:

• If both those show the delay symptom we are surely talking about a very crude network issue that is only seen in FOS/iPXE but not when the OS is booted - very unlikely. But if that’s the case you need to look into packet capturing as suggested before!!
• If the first test is instant but the second one is delayed we seem to have an iPXE issue - on the one hand I have never seen this on my HTTPS setups but also this is the most likely outcome from my perspective.
• If the first one is delayed but the second one gets an instant response - kind of impossible - then I have no idea and we need to re-think the whole case.
• And finally, if both tests yield in an instant response I would be puzzled as well. Then we’d need to dig into the differences between manual test and the normal PXE booting sequence.

[DBCountMan]

@Sebastian-Roth I pm’d you a pcap

Ran these tests on my hyper-v and xcp vms:

1. In the FOG debug console (Both Hyper-V and XCP showed this result)

wget --no-check-certificate https://fogserverip/fog/service/ipxe/bzImage
wget: not an http or ftp url: https://fogserverip/fog/service/ipxe/bzImage

2. kernel bzImage took about 3-4 seconds on hyper-v, 10 seconds on xcp, then returned with

bzImage...ok

[Sebastian Roth]

@brakcounty said in ipxe boot slow after changing to HTTPS:

wget: not an http or ftp url: https://fogserverip/fog/service/ipxe/bzImage

I have to admit that I have not tried it myself yet but I’d be pretty amazed if the wget binary we ship is not able to handle the HTTPS protocol. Anyhow, can you try curl -v -k https://fogserverip/fog/service/ipxe/bzImage instead?

kernel bzImage took about 3-4 seconds on hyper-v, 10 seconds on xcp, then returned with

Is this slower or faster than you see when PXE booting into a task?

I pm’d you a pcap

The first TCP SYN send by the client to open the connection should be answered by a SYN,ACK by the server but in the PCAP we see a simple ACK which wireshark tells us is “ACKed unseen segment” - like a packet from a different connection (but on the same ports!). This is very unusual! Then the client re-sends the initial SYN packet and gets a proper SYN,ACK back, returns an ACK to properly finish the TCP three way handshake.

Beside this strange behavior I wonder where the delay would happen. The first 9-10 seconds take for the DHCP DORA. The TCP handshake starts at 9.88 and goes straight into the SSL session setup. Between “Server Key Exchange, Server Hello Done” and “Client Key Exchange” there is a 2.5 second delay (caused by the client waiting) which I don’t find normal. Though I can imagine this is due to crypto algorithm calculations. The rest of the TCP communication looks to be fast.