Fog client installation error - Cannot install CA certificate

[jonhwood360]

Hello,

I’m attempting to setup a test lab for FOG, and am having no luck with the client install on Windows 10. I have tried searching this forum for info, and haven’t found any solutions that have worked for me yet.

Fog Server:
Fog 1.5.9
Ubuntu 20.04 Server LTS

Installed using HTTPS option during guided setup.

Windows 10
Windows 10 version 2004

Currently booted into Audit Mode for Configuration.

So I’m attempting to install the client using the SmartInstaller, with it always failing unable to install the CA certificate. Installation log is as follows:

Here is the Certificate itself:

I am able to download the certificates manually. I am also able to install the client without checking the “HTTPS” option, and then alter the operating mode to https in the service config json file, however that seems to start another problem with downloading another certificate (Middleware::Communication Download: https://10.40.40.5/fog/management/other/ssl/srvpublic.crt) and being unable to do so. I’ve also downloaded and manually installed the CA cert into the machine’s trusted root certificate store with no effect.

A side note, the smartinstaller download and execution triggered windows defender smartscreen.

Any ideas?

[Sebastian Roth]

@jonhwood360 Any chance you can disable defender just to see if that makes a difference?

[jonhwood360]

@sebastian-roth Thanks for the suggestion. I have disabled defender completely (via local GPO setting) and it made no difference in being able to run.

[Sebastian Roth]

@jonhwood360 Do you have a GPO forcing an install of a particular certificate in place? Once we had someone in the forums post that this kind of GPO prevented the fog-client from installing the CA cert. Though I would expect a different error message then.

And it’s probably not a firewall issue as you are able to manually download the cert files from that very same computer, right? Though you might still check the Apache logs on your FOG server if it seems to properly answer the request. Prepare the fog-client installer but don’t go to the last step where it would download the cert. Now in your FOG server console run: tail -f /var/log/apache2/*.log

Press ENTER twice and leave that command waiting while you quickly finish the fog-client installer. Now stop the tail command in the server (ctrl-c) and post all the new output lines that were added after you hit ENTER.

[jonhwood360]

@sebastian-roth:

1.) @jonhwood360 Do you have a GPO forcing an install of a particular certificate in place?
1a.) Nope, I do not.

2.) And it’s probably not a firewall issue as you are able to manually download the cert files from that very same computer, right?
2a.) Correct I can download the cert manually through a web browser on the computer I am attempting to install the client on.

3.) Though you might still check the Apache logs on your FOG server if it seems to properly answer the request. Prepare the fog-client installer but don’t go to the last step where it would download the cert. Now in your FOG server console run: tail -f /var/log/apache2/*.log. Press ENTER twice and leave that command waiting while you quickly finish the fog-client installer. Now stop the tail command in the server (ctrl-c) and post all the new output lines that were added after you hit ENTER.

3a.) See screenshot(s) of apache logs below.

Pre install

Failure

The log was from my other windows box on the network that has the fog admin console open. It does not appear that the installer is even reaching out to fogserver for the license. There is no firewall between the two VMs.

Thanks for helping troubleshoot this.

[Sebastian Roth]

@jonhwood360 I am sure we don’t see the request in the logs because it fails to establish the HTTPS connection in the first place. This goes along with the error you posted initially “The request was aborted: Could not create SSL/TLS secure channel”.

I expect you are using the fog-client version 0.12.0 that comes with the FOG server 1.5.9, right?

Possibly some .NET update broke our client lately?! When initially installing the fog-client we make it ignore that it doesn’t know the SSL CA yet (see code on github). So I could imagine some .NET update code changed the behavior. But on the other hand you said:

I’ve also downloaded and manually installed the CA cert into the machine’s trusted root certificate store with no effect.

In this case the SSL trust relationship should be all right with the CA (manually) installed and it would not need to rely on that code mentioned above.

I have to say that I have not tested on fully patched Windows 10 2004 lately but I can do so.

I might provide a binary with more debug output enabled for you to test and get more information. But will need a bit of time for that.

[Sebastian Roth]

@jonhwood360 Did you manually edit the Apache configuration or left it as generated by the FOG installer?

[jonhwood360]

@sebastian-roth

I expect you are using the fog-client version 0.12.0 that comes with the FOG server 1.5.9, right?

Yes

Possibly some .NET update broke our client lately?! When initially installing the fog-client we make it ignore that it doesn’t know the SSL CA yet ([see code on github] https://github.com/FOGProject/zazzles/blob/master/Zazzles/Middleware/Communication.cs#L300)). So I could imagine some .NET update code changed the behavior. But on the other hand you said:

I’ve also downloaded and manually installed the CA cert into the machine’s trusted root certificate store with no effect. In this case the SSL trust relationship should be all right with the CA (manually) installed and it would not need to rely on that code mentioned above.

I have tried manually importing the CA certificate and rerunning the install, and it fails at the same task. If you’d like I can retry this and screenshot the logs?

I have to say that I have not tested on fully patched Windows 10 2004 lately but I can do so.

I might provide a binary with more debug output enabled for you to test and get more information. But will need a bit of time for that.

Again, any assistance is greatly appreciated!

[Sebastian Roth]

@jonhwood360 said in Fog client installation error - Cannot install CA certificate:

I have tried manually importing the CA certificate and rerunning the install, and it fails at the same task. If you’d like I can retry this and screenshot the logs?

I might have mixed things up a bit and explained too little. What I meant is: in your first post you mentioned installing without HTTPS enabled (which seems to work), then change settings.json to enable HTTPS and start the FOGService. This will fail on authenticating against the FOG server when trying to load srvpublic.crt, correct?

As a workaround for now you can run the fog-client over HTTP. The communication protocol used by the fog-client will encrypt all information using state of the art certificate based crypto anyway. This encryption is part of the fog-client since a couple of years and still in place even if you use HTTPS - which then would be a double encrypted channel really.

[Sebastian Roth]

@jonhwood360 I just quickly tested on Windows 10 2004 (latest updates installed including 2021-01 .NET updates) and it installs and downloads the certs just fine.

I know this is not of much help to you yet but from that I would expect this to not be a general issue for everyone.

As I said, I will try to add some more debug output and compile a custom installer for you to test - probably not today though.

Just another question that came to my mind. You use the SmartInstaller.exe. Have you tried the MSI yet? Essentially the SmartInstaller has the MSI included, will extract it and call msiexec to install it. So there should be really no difference at all but please give it a try to make sure we see the same issue with both.

[jonhwood360]

@sebastian-roth said in Fog client installation error - Cannot install CA certificate:

@jonhwood360 I just quickly tested on Windows 10 2004 (latest updates installed including 2021-01 .NET updates) and it installs and downloads the certs just fine.

I know this is not of much help to you yet but from that I would expect this to not be a general issue for everyone.

As I said, I will try to add some more debug output and compile a custom installer for you to test - probably not today though.

Just another question that came to my mind. You use the SmartInstaller.exe. Have you tried the MSI yet? Essentially the SmartInstaller has the MSI included, will extract it and call msiexec to install it. So there should be really no difference at all but please give it a try to make sure we see the same issue with both.

Yes I have tried the MSI as well. I’ve tried running the smartinstaller as administrator, and installing the msi from an elevated command prompt as well.

I too am surprised about this. I wonder if this is a function of it the computer being in Audit mode (ctrl-shift-F3 at OS first boot right after install from media)?

[Sebastian Roth]

@jonhwood360 said in Fog client installation error - Cannot install CA certificate:

I wonder if this is a function of it the computer being in Audit mode (ctrl-shift-F3 at OS first boot right after install from media)?

Hmmm, I am not much of a Windows wiz, so can’t say. Would you have an idea @george1421 if that is possible?

[jonhwood360]

@sebastian-roth

So as a test I manually installed the certificate into the certificate store. I confirmed it was in fact installed through the certificate snapin in mmc. When I try to install the client, the certificate disappears from the store once it says it can’t install the certificate.

Pre-install

Post-Install

[george1421]

@sebastian-roth said in Fog client installation error - Cannot install CA certificate:

Would you have an idea @george1421 if that is possible?

I’m not sure how much help I can be, because we haven’t used the fog client in over 5 years. When we did use it we would load the service using MDT and then stop and disable the service right away just after it was installed. Then after sysprep and cloning we would restart it in the setupcompleted.cmd. We never touched audit mode because MDT did that part for us. We did use the MSI with command line parameters to install the fog client back then.

So one might wonder what the fog client uses to download the certificate? curl? Could MS have deprecated what the fog client uses to download files?

[Sebastian Roth]

@george1421 said in Fog client installation error - Cannot install CA certificate:

So one might wonder what the fog client uses to download the certificate? curl? Could MS have deprecated what the fog client uses to download files?

The fog-client uses WebClient.DownloadFile() - an officially provided function within the System.Net namespace provided by MS.

A quick search on the web didn’t reveal much about audit mode behaving differently with .NET calls or the cert store. Though I don’t know enough about it…

@jonhwood360 True, the fog-client installer will remove any cert from the store named “FOG Server CA” it finds before it loads the current one from the server to install that. It’s a way of making sure the right CA cert is being installed even if there are left overs from an old install.

Did you manually edit the Apache configuration or left it as generated by the FOG installer?

[Sebastian Roth]

@jonhwood360 Quickly added some debugging output and compiled a fresh MSI for you: https://github.com/FOGProject/fog-client/releases/download/0.12.0/FOGService_debug_CAcert.msi

This is not an official build but it will do a good job finding out what’s going wrong in your case I hope. Try installing with that MSI and then check the FOGService.install.log again. You should see more output in there than you had before. Post the new log output here.

[jonhwood360]

@sebastian-roth said in Fog client installation error - Cannot install CA certificate:

Did you manually edit the Apache configuration or left it as generated by the FOG installer?

No I did not.

[jonhwood360]

@sebastian-roth said in Fog client installation error - Cannot install CA certificate:

@jonhwood360 Quickly added some debugging output and compiled a fresh MSI for you: https://github.com/FOGProject/fog-client/releases/download/0.12.0/FOGService_debug_CAcert.msi

This is not an official build but it will do a good job finding out what’s going wrong in your case I hope. Try installing with that MSI and then check the FOGService.install.log again. You should see more output in there than you had before. Post the new log output here.

Here you go.

[Sebastian Roth]

@jonhwood360 Didn’t expect it to bail out that early. So it doesn’t even get to where I expected it to fail (SSL/TLS cert validity check).

Could you please try installing the fog-client on a system that is not in audit mode? Just want to make sure this has no effect.

The other thing we might take a look at is a network packet capture. Get the fog-client setup ready to the same point as last time when we looked at the Apache log files. Then run the following commands on your FOG server:

sudo -i
apt install tcpdump
tcpdump -nn -w /tmp/ssl.pcap host 10.40.40.14

Make sure you put in the IP address of the host you are trying to install fog-client on. Now leave the command sit there and finish the fog-client setup. After it failed, stop tcpdump (ctrl-c) and use WinSCP (or another secure copy tool) to copy the binary file /tmp/ssl.pcap over to another computer. Upload to any filesharing service you have access to and post a link here.

[jonhwood360]

@sebastian-roth said in Fog client installation error - Cannot install CA certificate:

@jonhwood360 Didn’t expect it to bail out that early. So it doesn’t even get to where I expected it to fail (SSL/TLS cert validity check).

Could you please try installing the fog-client on a system that is not in audit mode? Just want to make sure this has no effect.

The other thing we might take a look at is a network packet capture.

Here is the packet capture:
https://drive.google.com/file/d/1KM4WAsPPF43tVDomDUuR_HOEU_4bZ6oB/view?usp=sharing