SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.

[maverickws]

Hi all,

We’ve been looking to replace an old PXE server and we stumbled across the FOG Project which we considered quite interesting from what we were able to see.
We’ve implemented a server (RockyLinux 8.5) no problem everything is running just fine.
The particular issue we’re facing is the following:

Machine boot, gets undionly.kpxe, etc etc.
Then we get to:

:: Downloading 'http://server.ip/iso/rescue/airootfs.sfs'
curl: (60) SSL certificate problem: self signed certificate in certificate chain.
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.
ERROR: Downloading 'http://server.ip/iso/rescue/airootfs.sfs'
  Failing back to interactive prompt
  You can try to fix the problem normally, log out when you are finished
sh: can't access tty: job control turned off

The certificate is self-signed, yes. But if curl can’t go past by a self-signed certificate on a closed setup, how can we ever boot from FOG’s http server (suing SSL), being this a pre-boot environment?

If I were to disable SSL from FOG now - which isn’t an approach we fancy -, what would be the correct approach?
Thank you.

[Sebastian Roth]

@maverickws Can you please let us know what exactly you are trying to PXE boot here. The output posted is definitely not from the FOG code as far as I know.

The interesting thing I find is the “problematic” URL in the output posted starts with http:// (no SSL really)?!?

[george1421]

@maverickws Yes we need to see your params section of the custom ipxe boot menu. None of these error messages seem to be generated by FOG.

If your old pxe server is based on pxelinux/syslinux please provide the menu here.

Lastly did you enable https when you installed the fog server? That will help us debug if its an ssl error with the fog server or not.

[maverickws]

This post is deleted!

[maverickws]

@george1421 @Sebastian-Roth

Thank you both for your replies.

Our old PXE is dead and gone (hardware failed). It was an old project from someone who’s no longer with us, and it booted a collection of linux tools for rescue which is what we’re after with FOG. Now we’re moving this to a VM (Xen).

So we looked into SystemRescueCD and that’s what we’re trying to boot.
I’ve followed this guide on the FOG Project’s forum, what differs slightly are the paths (eg. instead of /var/www/utils/systemrescuecd we have /var/www/utils/sysresccd/boot/ and sysresccd/boot/x86_64) more respective to the original paths.

I did enable HTTPS when I installed the fog server.

I also noticed the http:// on the URL, but I’m figuring since I enabled https in the FOG server it’s redirecting curl from http to https despite showing the link on the message like that.

Please let know of any further info you may need. Thank you.

[george1421]

@maverickws I would have to check, but I think with fog if you enable https (port 443), http (port 80) is not configured. You could go into the apache config and enable a listener on port 80 if you want. Understand this is outside the scope of FOG, but heck that is what “opensource” is for (taking one project and twisting it into something different). You can see if http is listening with netstat -an | grep :80 command. If it returns something like 0.0.0.0 :80 Listening then port 80 (http) is enabled in apache.

When FOG ssl is enabled, that also embeds the matching certificate into the ipxe binaries. That way ipxe can call https URIs.

This next bit is speculation, but if curl is complaining about a self signed certificate, its possible that the root CA certificate that FOG creates needs to be moved into the trusted root certificate chain so that curl will trust the certificate presented by the apache server.

I had to do a similar concept to add a certificate for https inspection in our transparent proxy with these commands.

# cp ca_root_pub.cer /usr/local/share/ca-certificates/ca_root_pub.crt

# key needs to be a .crt file extension 

# update-ca-certificates
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

# restart apache

I’m not sure if that helps get you going but I think getting the certificates in the right place will help.

And for booting random stuff with FOG, I’m sure you’ve seen this tutorial: https://forums.fogproject.org/topic/10944/using-fog-to-pxe-boot-into-your-favorite-installer-images/8?_=1645017750532 There was a recent update of another fog admin that got a current release of Hirens to boot with FOG.

[maverickws]

@george1421
Once again I appreciate your feedback!

I’m just going to paste the results here. From netstat you’d say “port 80 isn’t listening for ipv4” but that’s not the case, I’ll demonstrate:

[root@fog ~]# netstat -an | grep :80
tcp6       0      0 :::80                   :::*                    LISTEN     
[root@fog ~]# netstat -an | grep http
unix  2      [ ACC ]     STREAM     LISTENING     28078    /etc/httpd/run/cgisock.886

the first line is for tcp6 so it’s listening for the IPv6 stack, apparently you’d say from this output it isn’t listening.
However, if I type http://fog.servers.domain.io I am redirected to https. So port 80 MUST be listening otherwise it wouldn’t redirect and I would simply get an error.

So I “curled” from my machine:

% curl -Lvv http://fog.servers.domain.io 
*   Trying 10.0.0.1:80...
* Connected to fog.servers.domain.io (10.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: fog.servers.domain.io
> User-Agent: curl/7.77.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Wed, 16 Feb 2022 13:35:22 GMT
< Server: Apache/2.4.37 (rocky) OpenSSL/1.1.1k
< Location: https://fog.servers.domain.io//
< Content-Length: 208
< Content-Type: text/html; charset=iso-8859-1
< 
* Ignoring the response-body
* Connection #0 to host fog.servers.domain.io left intact
* Issue another request to this URL: 'https://fog.servers.domain.io//'
*   Trying 10.0.0.1:443...
* Connected to fog.servers.domain.io (10.0.0.1) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 1
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I can’t find ca_root_pub.cer what should its location be? Also there is no path /usr/local/share/ca-certificates/
All fog certificates are at /var/www/html/fog/management/other (or are we talking about the certs in /etc/ssl ?)

[root@fog other]# ls -laR
.:
total 52
drwxr-xr-x.  3 apache apache    91 Feb 14 20:39 .
drwxr-xr-x. 10 apache apache   143 Feb 14 20:39 ..
-rw-r--r--.  1 apache apache  1301 Feb 14 20:39 ca.cert.der
-rwxr-xr-x.  1 apache apache  1818 Feb 14 20:39 ca.cert.pem
-rw-r--r--.  1 apache apache 35141 Feb 14 20:39 gpl-3.0.txt
-rw-r--r--.  1 apache apache  6152 Feb 14 20:39 index.php
drwxr-xr-x.  2 apache apache    27 Feb 14 20:39 ssl

./ssl:
total 4
drwxr-xr-x. 2 apache apache   27 Feb 14 20:39 .
drwxr-xr-x. 3 apache apache   91 Feb 14 20:39 ..
-rw-r--r--. 1 apache apache 1749 Feb 14 20:39 srvpublic.crt

About the topic with Hiren’s CD I did take a superficial look at it, actually there’s a number of other interesting tutorials here in the forum I’ve seen a few. Hiren’s doesn’t mean much to me as we don’t use Windows machines. Desktops/workstations are Apple, servers are linux (cli-only).
From what I can tell, the booting random stuff with fog as you called it works fine with this configuration for SystemRescueCD, it finds the pxe, downloads undionly.kpxe, moves on to boot the files for sysresccd.

What invokes curl instead of wget for example? Isn’t it the fog boot environment?

thank you.

[george1421]

@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:

I can’t find ca_root_pub.cer what should its location be?

That was an example of the root certificate I needed to import so that my proxy ssl inspection worked (not related to FOG). The path and directory structure was for debian 10. If you are using something else (though it looks from your apache path debian based) (edit: I just looked at your curl output closer and you are using rhel based system) you will need to use what is proper for your OS. This was just an example to get you started.

FWIW, I don’t use https with my fog server so I’m only guessing that port 80 is not configured. The netstat command will prove it one way or another. I can look into it, but I don’t think fog does a http->https redirect, but I could be wrong.

[george1421]

@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:

works fine with this configuration for SystemRescueCD

First it was my mistake equating SystemRescueCD with Hirens. I read very quickly and made the inaccurate connection.

As for using curl, I’m not sure what you are truly doing here. I’m interested in seeing if you can get curl to trust the srvpublic.crt file you mentioned that way it can draw files from the FOG server. But in my tutorial I’ve never used curl since iPXE will do what I need.

Back to my original post what does your param block look like or what method worked to download the SystemRescueCD in the past, we may need to translate it over to iPXE command format.

[maverickws]

@george1421 I used a RockyLinux minimal install, installed the Development Tools (dnf groupinstall Development Tools) and the epel repo.

No packages were manually installed. Everything came from the FOG installer. (used git to clone the repo, master branch)

I’m trying to understand which curl is being used to download the images. What exactly is the certificate store that curl is using?

FOG is listening on port 80 and IS redirecting to https.
This is the http section of my fog.conf from /etc/httpd/conf.d

<VirtualHost *:80>
    <FilesMatch "\.php$">
        SetHandler "proxy:fcgi://127.0.0.1:9000/"
    </FilesMatch>
    ServerName 10.0.0.1
    ServerAlias fog.servers.domain.io
    DocumentRoot /var/www/html/
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]
    RewriteRule /management/other/ca.cert.der$ - [L]
    RewriteCond %{HTTPS} off
    RewriteRule (.*) https://%{HTTP_HOST}/$1 [R,L]
</VirtualHost>

As for using curl, I’m not sure what you are truly doing here. I’m interested in seeing if you can get curl to trust the srvpublic.crt file you mentioned that way it can draw files from the FOG server. But in my tutorial I’ve never used curl since iPXE will do what I need.

Back to my original post what does your param block look like or what method worked to download the SystemRescueCD in the past, we may need to translate it over to iPXE command format.

About curl, I’m not sure either. That’s really what I was trying to figure out.
Because I don’t think that the curl being used is binary from the FOG server. So I don’t see how it would either way trust that certificate. I am thinking the curl is loaded from the boot image loaded by FOG.

The param block looks exactly like the tutorial, only the paths after {fog-ip} have been edited to match our env, which as I mentioned is simply instead of SystemRescueCD its sysresccd /boot /x86_64 respecting the original files locations on the CD.

I believe that if I hadn’t enabled SSL the boot would be working just fine actually.

[george1421]

@maverickws OK lets take a step back from the edge here <joke>

I just downloaded the SystemRescueCD iso image. I don’t have a lot of time at the moment to test this, but I see what they are doing and I found they are using syslinx syntax.

This is the config file of interest

LABEL sysresccd_nbd
TEXT HELP
Boot the SystemRescue live medium (Using NBD).
It allows you to install Arch Linux or perform system maintenance.
ENDTEXT
MENU LABEL Boot SystemRescue (NBD)
LINUX boot/x86_64/vmlinuz
INITRD boot/intel_ucode.img,boot/amd_ucode.img,boot/x86_64/sysresccd.img
APPEND archisobasedir=sysresccd archisolabel=RESCUE901 archiso_nbd_srv=${pxeserver} iomem=relaxed
SYSAPPEND 3

LABEL sysresccd_nfs
TEXT HELP
Boot the SystemRescue live medium (Using NFS).
It allows you to install Arch Linux or perform system maintenance.
ENDTEXT
MENU LABEL Boot SystemRescue (NFS)
LINUX boot/x86_64/vmlinuz
INITRD boot/intel_ucode.img,boot/amd_ucode.img,boot/x86_64/sysresccd.img
APPEND archisobasedir=sysresccd archiso_nfs_srv=${pxeserver}:/run/sysresccd/bootmnt iomem=relaxed
SYSAPPEND 3

LABEL sysresccd_http
TEXT HELP
Boot the SystemRescue live medium (Using HTTP).
It allows you to install Arch Linux or perform system maintenance.
ENDTEXT
MENU LABEL Boot SystemRescue (HTTP)
LINUX boot/x86_64/vmlinuz
INITRD boot/intel_ucode.img,boot/amd_ucode.img,boot/x86_64/sysresccd.img
APPEND archisobasedir=sysresccd archiso_http_srv=http://${pxeserver}/ iomem=relaxed
SYSAPPEND 3

This shows 3 different methods of starting up the systemrescue cd. One by pxe (tftp), one by nfs, and one by http.

So lets take this syslinux syntax and convert it over it iPXE (what fog uses)

LABEL sysresccd_nbd
TEXT HELP
Boot the SystemRescue live medium (Using NBD).
It allows you to install Arch Linux or perform system maintenance.
ENDTEXT
MENU LABEL Boot SystemRescue (NBD)
LINUX boot/x86_64/vmlinuz
INITRD boot/intel_ucode.img,boot/amd_ucode.img,boot/x86_64/sysresccd.img
APPEND archisobasedir=sysresccd archisolabel=RESCUE901 archiso_nbd_srv=${pxeserver} iomem=relaxed
SYSAPPEND 3

The translated params section of the fog iPXE menu would look like this: understand this below will not work out of the box because the files are not in the right places on the FOG server. But in general this is the flow

kernel tftp://${fog-ip}/boot/x86_64/vmlinuz
imgfetch  tftp://${fog-ip}/boot/intel_ucode.img
imgfetch  tftp://${fog-ip}/boot/amd_ucode.img
imgfetch  tftp://${fog-ip}/boot/x86_64/sysresccd.img
imgargs vmlinuz archisobasedir=sysresccd archisolabel=RESCUE901 archiso_nbd_srv=${fog-ip} iomem=relaxed
boot || goto MENU

There are examples of how this is done on that monster tutorial I referenced before (tftp, nfs, and http image loading).

As I said I don’t have time at the moment to work out the exact step by step but it surely looks like a possible thing.

I do have a reference to systemrescue on that tutorial page here: https://forums.fogproject.org/post/112052 Its from 2018. It looks like a similar program.

[maverickws]

@george1421 Please allow me to thank you for your time and effort following up on this topic.

I had seen that tutorial also, the problem with it is that references a quite old version of SystemRescue (5.x) while we’re currently at 9.01 and there are many critical differences between them. The tutorial for SystemRescue 8.1 is very similar tho, that’s why I followed it.

So basically what you’re suggesting is that, instead of http I use the network block device method.
From what I’m able to see, no NBD server is installed with the FOG server, do you confirm?

I understand that this is a workaround proposition to get SystemRescueCD to work/boot. But at the same time we’re digressing from the original issue, which is CURL and it’s trust chain. If I apply this workaround, I’ll still be unable to use http and will have to always resort to nbd, AND will have to configure the nbd service first, correct?

I had an alternative method in mind, which would be adding a rule to disable the https redirect for specific paths. (like when trying to reach anything within /utils)

[george1421]

@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:

But at the same time we’re digressing from the original issue, which is CURL and it’s trust chain. If I apply this workaround,

I’ll have to spin up a fog server in my dev environment when I get home tonight to test https and http redirects. Since I haven’t needed to do this before I can’t say off the top of my head. I think there is an answer to this issue if I can spend some time thinking it through. At the very least we can add a new virtual site to the apache configuration on a different port than 80 to do what you want, but again I haven’t put too much thought into it at the moment.

[Sebastian Roth]

@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:

About curl, I’m not sure either. That’s really what I was trying to figure out.
Because I don’t think that the curl being used is binary from the FOG server. So I don’t see how it would either way trust that certificate. I am thinking the curl is loaded from the boot image loaded by FOG.

I have to admit I have not had the time to play with it but with some good background in FOG and PXE booting distros I am sure it’s curl included in the SystemRescueCD ISO.

So working around this you have a few options:

• Disable HTTP->HTTPS redirect for certain URLs
• Extract the ISO, add --insecure to the curl call and re-master the ISO -> probably a fair amount of work, I haven’t done it before
• See if you can PXE boot it via NFS rather than HTTP

[maverickws]

@george1421 I have time actually this is not the most urgent matter at hand. (and hopefully I won’t get struck by the pressing need of it in the next days).

I think I’d rather have it configured with a <Directory> block inside the VirtualHost preventing the HTTPS redirect than making a new virtual site to apache listening on another port. This is just as I feel this is a simpler option.

I’ve been trying a few stances but haven’t got it to work it.

I have a question though:
The boot process is:

iPXE gets undionly.kpxe;
configures devices,
downloads:

tftp://10.0.0.1/default.ipxe
https://10.0.0.1/fog/service/ipxe/boot.php
https://10.0.0.1/fog/service/ipxe/bg.png

And it loads the boot menu.
So here we’ve used https without any issue, I dunno what util is being used for the download (curl, wget, other).

I select SystemRescueCD 9.01 from the menu, it downloads:

tftp://10.0.0.1/utils/sysresccd9.01/boot/x86_64/vmlinuz
tftp://10.0.0.1/utils/sysresccd9.01/boot/intel_ucode.img
tftp://10.0.0.1/utils/sysresccd9.01/boot/amd_ucode.img
tftp://10.0.0.1/utils/sysresccd9.01/boot/x86_64/sysresccd.img
docache...
Probing EDD (edd=off to disable)... ok
[    3.482042] initramfs unpacking failed: invalid magic at start of compressed archive
:: running early hook [udev]
Starting version 250.3-3-arch
:: running early hook [archiso_pxe_nbd]
:: running hook [udev]
:: Triggering uevents...
[    3.739314] vid-5659: 19 xenbus_dev_probe on device/vbd/5696
:: running hook [memdisk]
:: running hook [findroot]
:: running hook [archiso]
:: running hook [archiso_loop_mnt]
:: running hook [archiso_pxe_common]
Attempting to configure network interface eth0 ...
IP-Config: eth0 hardware address 2e:00:7f:d3:ab:3c mtu 1500 DHCP
IP-Config: eth0 guessed broadcast address 10.0.0.255
IP-Config: eth0 complete (from 10.0.0.252):
 address: 10.0.0.70        broadcast: 10.0.0.255       netmask: 255.255.255.0
 gateway: 10.0.0.254       dns0     : 10.0.0.254       dns1   : 0.0.0.0
 host   : testsrv01
 domain : servers.domain.io
 rootserver: 10.0.0.1 rootpath:
 filename  : undionly.kpxe
SUCCESS: Network interface eth0 has been successfully configured
:: running hook [archiso_pxe_nbd]
:: running hook [archiso_pxe_http]
:: running hook [archiso_pxe_nfs]
:: running hook [encrypt]
:: Mounting /run/archiso/httpspace (temps) filesystem, size='75%'
Downloading 'http://10.0.0.1/utils/sysresccd9.01/sysresccd/x86_64/airootfs.sfs'
  % Total    % Received % Xferd  Average Speed   Time   Time    Time  Current
                                 Dload  Upload   Total  Spent   Left  Speed
100   253  100   253    0     0   2555      0 --:--:-- --:--:-- --:--:-- 2581
curl: (60) SSL certificate problem: self signed certificate in certificate chain.
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.
ERROR: Downloading 'http://fog.servers.domain.io/utils/sysresccd9.01/sysresccd/x86_64/airootfs.sfs'
  Failing back to interactive prompt
  You can try to fix the problem normally, log out when you are finished
sh: can't access tty: job control turned off
[rootfs ]#

My question is, is this “rootfs” loaded from FOG, or from the files loaded already from sysresccd?
I have been assuming this is from the SystemRescueCD image, as initially there was no issue for iPXE to get to https:// … boot.php.

[Sebastian Roth]

@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:

My question is, is this “rootfs” loaded from FOG, or from the files loaded already from sysresccd?

The later I am sure. See my post an hour ago.

[maverickws]

@sebastian-roth Hi there thank you too for following up on this!

Sorry I guess I missed your reply while I was replying to george!

I was thinking too that it was SRCD’s curl, makes sense given the evidence so far.
Currently I’m going for the first suggested option:

• Disable the HTTP->HTTPS redirect for certain URLs

I have made a test only but haven’t been successful to put it to work. Basically I’ve tested adding a line to the VirtualHost Block below RewriteEngine On:

RewriteEngine On
    RewriteRule ^(utils)($|/) - [L]
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]
    RewriteRule /management/other/ca.cert.der$ - [L]
    RewriteCond %{HTTPS} off
    RewriteRule (.*) https://%{HTTP_HOST}/$1 [R,L]

Trying to figure this one out better.

[george1421]

@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:

Probing EDD (edd=off to disable)... ok
[    3.482042] initramfs unpacking failed: invalid magic at start of compressed archive
:: running early hook [udev]

Right here is where vmlinuz takes over control of the target computer. At this point FOG and ipxe is out of the picture. So the rescue CD is in control of the workstation. Also the curl command is coming from inside the initrd (i assume sysresccd.img) so anything you do to fog (outside the apache settings) is out of the picture. The root CA certificate will need to be added to the sysresccd.img file since that is where curl is coming from.

In my opinion, I would turn off https on the fog server so this app will play better, but do what fits your organization best.

[maverickws]

SOLVED!

Solved it by adding the line RewriteCond %{REQUEST_URI} !^/utils to the VirtualHost block at /etc/httpd/conf.d/fog.conf so it looks like this now:

<VirtualHost *:80>
    <FilesMatch "\.php$">
        SetHandler "proxy:fcgi://127.0.0.1:9000/"
    </FilesMatch>
    ServerName 10.0.0.1
    ServerAlias fog.servers.domain.io
    DocumentRoot /var/www/html/
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]
    RewriteRule /management/other/ca.cert.der$ - [L]
    RewriteCond %{HTTPS} off
    RewriteCond %{REQUEST_URI} !^/utils
    RewriteRule (.*) https://%{HTTP_HOST}/$1 [R,L]
</VirtualHost>

@george1421 hi and once again let me thank you for your time.
I just managed to get over the problem and disabled the HTTPS redirect for the /utils path.
SystemRescue loaded just fine.

I would suggest, maybe, adding this rule by default when we enable SSL on the fog server, and having therefor a specific path to put boot files that could be blocked by curl’s certificate check.