SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.
-
This post is deleted! -
Thank you both for your replies.
Our old PXE is dead and gone (hardware failed). It was an old project from someone who’s no longer with us, and it booted a collection of linux tools for rescue which is what we’re after with FOG. Now we’re moving this to a VM (Xen).
So we looked into SystemRescueCD and that’s what we’re trying to boot.
I’ve followed this guide on the FOG Project’s forum, what differs slightly are the paths (eg. instead of/var/www/utils/systemrescuecd
we have/var/www/utils/sysresccd/boot/
andsysresccd/boot/x86_64
) more respective to the original paths.I did enable HTTPS when I installed the fog server.
I also noticed the
http://
on the URL, but I’m figuring since I enabled https in the FOG server it’s redirecting curl from http to https despite showing the link on the message like that.Please let know of any further info you may need. Thank you.
-
@maverickws I would have to check, but I think with fog if you enable https (port 443), http (port 80) is not configured. You could go into the apache config and enable a listener on port 80 if you want. Understand this is outside the scope of FOG, but heck that is what “opensource” is for (taking one project and twisting it into something different). You can see if http is listening with
netstat -an | grep :80
command. If it returns something like0.0.0.0 :80 Listening
then port 80 (http) is enabled in apache.When FOG ssl is enabled, that also embeds the matching certificate into the ipxe binaries. That way ipxe can call https URIs.
This next bit is speculation, but if curl is complaining about a self signed certificate, its possible that the root CA certificate that FOG creates needs to be moved into the trusted root certificate chain so that curl will trust the certificate presented by the apache server.
I had to do a similar concept to add a certificate for https inspection in our transparent proxy with these commands.
# cp ca_root_pub.cer /usr/local/share/ca-certificates/ca_root_pub.crt # key needs to be a .crt file extension # update-ca-certificates Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. # restart apache
I’m not sure if that helps get you going but I think getting the certificates in the right place will help.
And for booting random stuff with FOG, I’m sure you’ve seen this tutorial: https://forums.fogproject.org/topic/10944/using-fog-to-pxe-boot-into-your-favorite-installer-images/8?_=1645017750532 There was a recent update of another fog admin that got a current release of Hirens to boot with FOG.
-
@george1421
Once again I appreciate your feedback!I’m just going to paste the results here. From netstat you’d say “port 80 isn’t listening for ipv4” but that’s not the case, I’ll demonstrate:
[root@fog ~]# netstat -an | grep :80 tcp6 0 0 :::80 :::* LISTEN [root@fog ~]# netstat -an | grep http unix 2 [ ACC ] STREAM LISTENING 28078 /etc/httpd/run/cgisock.886
the first line is for tcp6 so it’s listening for the IPv6 stack, apparently you’d say from this output it isn’t listening.
However, if I type http://fog.servers.domain.io I am redirected tohttps
. So port 80 MUST be listening otherwise it wouldn’t redirect and I would simply get an error.So I “curled” from my machine:
% curl -Lvv http://fog.servers.domain.io * Trying 10.0.0.1:80... * Connected to fog.servers.domain.io (10.0.0.1) port 80 (#0) > GET / HTTP/1.1 > Host: fog.servers.domain.io > User-Agent: curl/7.77.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 302 Found < Date: Wed, 16 Feb 2022 13:35:22 GMT < Server: Apache/2.4.37 (rocky) OpenSSL/1.1.1k < Location: https://fog.servers.domain.io// < Content-Length: 208 < Content-Type: text/html; charset=iso-8859-1 < * Ignoring the response-body * Connection #0 to host fog.servers.domain.io left intact * Issue another request to this URL: 'https://fog.servers.domain.io//' * Trying 10.0.0.1:443... * Connected to fog.servers.domain.io (10.0.0.1) port 443 (#1) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem * CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (OUT), TLS alert, unknown CA (560): * SSL certificate problem: self signed certificate in certificate chain * Closing connection 1 curl: (60) SSL certificate problem: self signed certificate in certificate chain More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.
I can’t find
ca_root_pub.cer
what should its location be? Also there is no path/usr/local/share/ca-certificates/
All fog certificates are at/var/www/html/fog/management/other
(or are we talking about the certs in/etc/ssl
?)[root@fog other]# ls -laR .: total 52 drwxr-xr-x. 3 apache apache 91 Feb 14 20:39 . drwxr-xr-x. 10 apache apache 143 Feb 14 20:39 .. -rw-r--r--. 1 apache apache 1301 Feb 14 20:39 ca.cert.der -rwxr-xr-x. 1 apache apache 1818 Feb 14 20:39 ca.cert.pem -rw-r--r--. 1 apache apache 35141 Feb 14 20:39 gpl-3.0.txt -rw-r--r--. 1 apache apache 6152 Feb 14 20:39 index.php drwxr-xr-x. 2 apache apache 27 Feb 14 20:39 ssl ./ssl: total 4 drwxr-xr-x. 2 apache apache 27 Feb 14 20:39 . drwxr-xr-x. 3 apache apache 91 Feb 14 20:39 .. -rw-r--r--. 1 apache apache 1749 Feb 14 20:39 srvpublic.crt
About the topic with Hiren’s CD I did take a superficial look at it, actually there’s a number of other interesting tutorials here in the forum I’ve seen a few. Hiren’s doesn’t mean much to me as we don’t use Windows machines. Desktops/workstations are Apple, servers are linux (cli-only).
From what I can tell, the booting random stuff with fog as you called it works fine with this configuration for SystemRescueCD, it finds the pxe, downloads undionly.kpxe, moves on to boot the files for sysresccd.What invokes curl instead of wget for example? Isn’t it the fog boot environment?
thank you.
-
@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:
I can’t find ca_root_pub.cer what should its location be?
That was an example of the root certificate I needed to import so that my proxy ssl inspection worked (not related to FOG). The path and directory structure was for debian 10. If you are using something else (
though it looks from your apache path debian based) (edit: I just looked at your curl output closer and you are using rhel based system) you will need to use what is proper for your OS. This was just an example to get you started.FWIW, I don’t use https with my fog server so I’m only guessing that port 80 is not configured. The netstat command will prove it one way or another. I can look into it, but I don’t think fog does a http->https redirect, but I could be wrong.
-
@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:
works fine with this configuration for SystemRescueCD
First it was my mistake equating SystemRescueCD with Hirens. I read very quickly and made the inaccurate connection.
As for using curl, I’m not sure what you are truly doing here. I’m interested in seeing if you can get curl to trust the srvpublic.crt file you mentioned that way it can draw files from the FOG server. But in my tutorial I’ve never used curl since iPXE will do what I need.
Back to my original post what does your param block look like or what method worked to download the SystemRescueCD in the past, we may need to translate it over to iPXE command format.
-
@george1421 I used a RockyLinux minimal install, installed the Development Tools (dnf groupinstall Development Tools) and the epel repo.
No packages were manually installed. Everything came from the FOG installer. (used git to clone the repo, master branch)
I’m trying to understand which curl is being used to download the images. What exactly is the certificate store that curl is using?
FOG is listening on port 80 and IS redirecting to https.
This is the http section of myfog.conf
from/etc/httpd/conf.d
<VirtualHost *:80> <FilesMatch "\.php$"> SetHandler "proxy:fcgi://127.0.0.1:9000/" </FilesMatch> ServerName 10.0.0.1 ServerAlias fog.servers.domain.io DocumentRoot /var/www/html/ RewriteEngine On RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) RewriteRule .* - [F] RewriteRule /management/other/ca.cert.der$ - [L] RewriteCond %{HTTPS} off RewriteRule (.*) https://%{HTTP_HOST}/$1 [R,L] </VirtualHost>
As for using curl, I’m not sure what you are truly doing here. I’m interested in seeing if you can get curl to trust the srvpublic.crt file you mentioned that way it can draw files from the FOG server. But in my tutorial I’ve never used curl since iPXE will do what I need.
Back to my original post what does your param block look like or what method worked to download the SystemRescueCD in the past, we may need to translate it over to iPXE command format.
About curl, I’m not sure either. That’s really what I was trying to figure out.
Because I don’t think that the curl being used is binary from the FOG server. So I don’t see how it would either way trust that certificate. I am thinking the curl is loaded from the boot image loaded by FOG.The param block looks exactly like the tutorial, only the paths after {fog-ip} have been edited to match our env, which as I mentioned is simply instead of
SystemRescueCD
itssysresccd
/boot
/x86_64
respecting the original files locations on the CD.I believe that if I hadn’t enabled SSL the boot would be working just fine actually.
-
@maverickws OK lets take a step back from the edge here <joke>
I just downloaded the SystemRescueCD iso image. I don’t have a lot of time at the moment to test this, but I see what they are doing and I found they are using syslinx syntax.
This is the config file of interest
LABEL sysresccd_nbd TEXT HELP Boot the SystemRescue live medium (Using NBD). It allows you to install Arch Linux or perform system maintenance. ENDTEXT MENU LABEL Boot SystemRescue (NBD) LINUX boot/x86_64/vmlinuz INITRD boot/intel_ucode.img,boot/amd_ucode.img,boot/x86_64/sysresccd.img APPEND archisobasedir=sysresccd archisolabel=RESCUE901 archiso_nbd_srv=${pxeserver} iomem=relaxed SYSAPPEND 3 LABEL sysresccd_nfs TEXT HELP Boot the SystemRescue live medium (Using NFS). It allows you to install Arch Linux or perform system maintenance. ENDTEXT MENU LABEL Boot SystemRescue (NFS) LINUX boot/x86_64/vmlinuz INITRD boot/intel_ucode.img,boot/amd_ucode.img,boot/x86_64/sysresccd.img APPEND archisobasedir=sysresccd archiso_nfs_srv=${pxeserver}:/run/sysresccd/bootmnt iomem=relaxed SYSAPPEND 3 LABEL sysresccd_http TEXT HELP Boot the SystemRescue live medium (Using HTTP). It allows you to install Arch Linux or perform system maintenance. ENDTEXT MENU LABEL Boot SystemRescue (HTTP) LINUX boot/x86_64/vmlinuz INITRD boot/intel_ucode.img,boot/amd_ucode.img,boot/x86_64/sysresccd.img APPEND archisobasedir=sysresccd archiso_http_srv=http://${pxeserver}/ iomem=relaxed SYSAPPEND 3
This shows 3 different methods of starting up the systemrescue cd. One by pxe (tftp), one by nfs, and one by http.
So lets take this syslinux syntax and convert it over it iPXE (what fog uses)
LABEL sysresccd_nbd TEXT HELP Boot the SystemRescue live medium (Using NBD). It allows you to install Arch Linux or perform system maintenance. ENDTEXT MENU LABEL Boot SystemRescue (NBD) LINUX boot/x86_64/vmlinuz INITRD boot/intel_ucode.img,boot/amd_ucode.img,boot/x86_64/sysresccd.img APPEND archisobasedir=sysresccd archisolabel=RESCUE901 archiso_nbd_srv=${pxeserver} iomem=relaxed SYSAPPEND 3
The translated params section of the fog iPXE menu would look like this:
understand this below will not work out of the box because the files are not in the right places on the FOG server. But in general this is the flow
kernel tftp://${fog-ip}/boot/x86_64/vmlinuz imgfetch tftp://${fog-ip}/boot/intel_ucode.img imgfetch tftp://${fog-ip}/boot/amd_ucode.img imgfetch tftp://${fog-ip}/boot/x86_64/sysresccd.img imgargs vmlinuz archisobasedir=sysresccd archisolabel=RESCUE901 archiso_nbd_srv=${fog-ip} iomem=relaxed boot || goto MENU
There are examples of how this is done on that monster tutorial I referenced before (tftp, nfs, and http image loading).
As I said I don’t have time at the moment to work out the exact step by step but it surely looks like a possible thing.
I do have a reference to systemrescue on that tutorial page here: https://forums.fogproject.org/post/112052 Its from 2018. It looks like a similar program.
-
@george1421 Please allow me to thank you for your time and effort following up on this topic.
I had seen that tutorial also, the problem with it is that references a quite old version of SystemRescue (5.x) while we’re currently at 9.01 and there are many critical differences between them. The tutorial for SystemRescue 8.1 is very similar tho, that’s why I followed it.
So basically what you’re suggesting is that, instead of http I use the network block device method.
From what I’m able to see, no NBD server is installed with the FOG server, do you confirm?I understand that this is a workaround proposition to get SystemRescueCD to work/boot. But at the same time we’re digressing from the original issue, which is CURL and it’s trust chain. If I apply this workaround, I’ll still be unable to use http and will have to always resort to nbd, AND will have to configure the nbd service first, correct?
I had an alternative method in mind, which would be adding a rule to disable the https redirect for specific paths. (like when trying to reach anything within
/utils
) -
@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:
But at the same time we’re digressing from the original issue, which is CURL and it’s trust chain. If I apply this workaround,
I’ll have to spin up a fog server in my dev environment when I get home tonight to test https and http redirects. Since I haven’t needed to do this before I can’t say off the top of my head. I think there is an answer to this issue if I can spend some time thinking it through. At the very least we can add a new virtual site to the apache configuration on a different port than 80 to do what you want, but again I haven’t put too much thought into it at the moment.
-
@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:
About curl, I’m not sure either. That’s really what I was trying to figure out.
Because I don’t think that the curl being used is binary from the FOG server. So I don’t see how it would either way trust that certificate. I am thinking the curl is loaded from the boot image loaded by FOG.I have to admit I have not had the time to play with it but with some good background in FOG and PXE booting distros I am sure it’s curl included in the SystemRescueCD ISO.
So working around this you have a few options:
- Disable HTTP->HTTPS redirect for certain URLs
- Extract the ISO, add
--insecure
to the curl call and re-master the ISO -> probably a fair amount of work, I haven’t done it before - See if you can PXE boot it via NFS rather than HTTP
-
@george1421 I have time actually this is not the most urgent matter at hand. (and hopefully I won’t get struck by the pressing need of it in the next days).
I think I’d rather have it configured with a <Directory> block inside the VirtualHost preventing the HTTPS redirect than making a new virtual site to apache listening on another port. This is just as I feel this is a simpler option.
I’ve been trying a few stances but haven’t got it to work it.
I have a question though:
The boot process is:iPXE gets undionly.kpxe;
configures devices,
downloads:tftp://10.0.0.1/default.ipxe https://10.0.0.1/fog/service/ipxe/boot.php https://10.0.0.1/fog/service/ipxe/bg.png
And it loads the boot menu.
So here we’ve used https without any issue, I dunno what util is being used for the download (curl, wget, other).I select SystemRescueCD 9.01 from the menu, it downloads:
tftp://10.0.0.1/utils/sysresccd9.01/boot/x86_64/vmlinuz tftp://10.0.0.1/utils/sysresccd9.01/boot/intel_ucode.img tftp://10.0.0.1/utils/sysresccd9.01/boot/amd_ucode.img tftp://10.0.0.1/utils/sysresccd9.01/boot/x86_64/sysresccd.img docache... Probing EDD (edd=off to disable)... ok [ 3.482042] initramfs unpacking failed: invalid magic at start of compressed archive :: running early hook [udev] Starting version 250.3-3-arch :: running early hook [archiso_pxe_nbd] :: running hook [udev] :: Triggering uevents... [ 3.739314] vid-5659: 19 xenbus_dev_probe on device/vbd/5696 :: running hook [memdisk] :: running hook [findroot] :: running hook [archiso] :: running hook [archiso_loop_mnt] :: running hook [archiso_pxe_common] Attempting to configure network interface eth0 ... IP-Config: eth0 hardware address 2e:00:7f:d3:ab:3c mtu 1500 DHCP IP-Config: eth0 guessed broadcast address 10.0.0.255 IP-Config: eth0 complete (from 10.0.0.252): address: 10.0.0.70 broadcast: 10.0.0.255 netmask: 255.255.255.0 gateway: 10.0.0.254 dns0 : 10.0.0.254 dns1 : 0.0.0.0 host : testsrv01 domain : servers.domain.io rootserver: 10.0.0.1 rootpath: filename : undionly.kpxe SUCCESS: Network interface eth0 has been successfully configured :: running hook [archiso_pxe_nbd] :: running hook [archiso_pxe_http] :: running hook [archiso_pxe_nfs] :: running hook [encrypt] :: Mounting /run/archiso/httpspace (temps) filesystem, size='75%' Downloading 'http://10.0.0.1/utils/sysresccd9.01/sysresccd/x86_64/airootfs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 253 100 253 0 0 2555 0 --:--:-- --:--:-- --:--:-- 2581 curl: (60) SSL certificate problem: self signed certificate in certificate chain. More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. ERROR: Downloading 'http://fog.servers.domain.io/utils/sysresccd9.01/sysresccd/x86_64/airootfs.sfs' Failing back to interactive prompt You can try to fix the problem normally, log out when you are finished sh: can't access tty: job control turned off [rootfs ]#
My question is, is this “rootfs” loaded from FOG, or from the files loaded already from sysresccd?
I have been assuming this is from the SystemRescueCD image, as initially there was no issue for iPXE to get to https:// … boot.php. -
@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:
My question is, is this “rootfs” loaded from FOG, or from the files loaded already from sysresccd?
The later I am sure. See my post an hour ago.
-
@sebastian-roth Hi there thank you too for following up on this!
Sorry I guess I missed your reply while I was replying to george!
I was thinking too that it was SRCD’s curl, makes sense given the evidence so far.
Currently I’m going for the first suggested option:- Disable the HTTP->HTTPS redirect for certain URLs
I have made a test only but haven’t been successful to put it to work. Basically I’ve tested adding a line to the VirtualHost Block below
RewriteEngine On
:RewriteEngine On RewriteRule ^(utils)($|/) - [L] RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) RewriteRule .* - [F] RewriteRule /management/other/ca.cert.der$ - [L] RewriteCond %{HTTPS} off RewriteRule (.*) https://%{HTTP_HOST}/$1 [R,L]
Trying to figure this one out better.
-
@maverickws said in SSL & Fog http - curl failed to verify the legitimacy of the server and therefor could not establish a secure connection to it.:
Probing EDD (edd=off to disable)... ok [ 3.482042] initramfs unpacking failed: invalid magic at start of compressed archive :: running early hook [udev]
Right here is where
vmlinuz
takes over control of the target computer. At this point FOG and ipxe is out of the picture. So the rescue CD is in control of the workstation. Also the curl command is coming from inside the initrd (i assume sysresccd.img) so anything you do to fog (outside the apache settings) is out of the picture. The root CA certificate will need to be added to the sysresccd.img file since that is where curl is coming from.In my opinion, I would turn off https on the fog server so this app will play better, but do what fits your organization best.
-
SOLVED!
Solved it by adding the line
RewriteCond %{REQUEST_URI} !^/utils
to the VirtualHost block at/etc/httpd/conf.d/fog.conf
so it looks like this now:<VirtualHost *:80> <FilesMatch "\.php$"> SetHandler "proxy:fcgi://127.0.0.1:9000/" </FilesMatch> ServerName 10.0.0.1 ServerAlias fog.servers.domain.io DocumentRoot /var/www/html/ RewriteEngine On RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) RewriteRule .* - [F] RewriteRule /management/other/ca.cert.der$ - [L] RewriteCond %{HTTPS} off RewriteCond %{REQUEST_URI} !^/utils RewriteRule (.*) https://%{HTTP_HOST}/$1 [R,L] </VirtualHost>
@george1421 hi and once again let me thank you for your time.
I just managed to get over the problem and disabled the HTTPS redirect for the/utils
path.
SystemRescue loaded just fine.I would suggest, maybe, adding this rule by default when we enable SSL on the fog server, and having therefor a specific path to put boot files that could be blocked by curl’s certificate check.