Fog 1.5.5 to 1.5.8 - Creating SSL Certificate - FAILED

JYost

The current fogstorage database password does not meet high
security standards. We will generate a new password and update
all the settings on this FOG server for you. Please take note
of the following credentials that you need to manually update
on all your storage nodes’ /opt/fog/.fogsettings configuration
files and re-run (!) the FOG installer:
snmysqluser=‘fogstorage’
snmysqlpass=‘fO7hK:KyRMUeKkF71Nho’

Press [Enter] to proceed after you noted down the credentials.

Setting up MySQL user and database…OK
Backing up user reports…Done
Stopping web service…OK
Setting up Apache and PHP files…OK
Testing and removing symbolic links if found…OK
Backing up old data…OK
Copying new files to web folder…OK
Creating config file…OK
Downloading kernel, init and fog-client binaries…Done
Extracting the binaries archive…OK
Copying binaries to destination paths…OK
Enabling apache and fpm services on boot…OK
Creating SSL Certificate…Failed!
root@usktfoglp001:~/fogproject/bin/error_logs#

JYost

As instructed to do I’ve edited my .fogsettings file as follows:

snmysqluser=‘fogstorage’
snmysqlpass=‘fO7hKyRMUeKkF71Nho’

(I had to change the permissions on this file as it was read only, thereafter I was able to save the above)

JYost

edit:

snmysqlpass=‘fO7hK:KyRMUeKkF71Nho’

Sebastian Roth

@JYost Sorry, should have been more clear on this. There is more than one log file in that error_logs directory. We need the content of the file called fog_error_1.5.8.log…

JYost

inflating: packages/clientfiles/SmartInstaller.exe
‘packages/kernels/bzImage’ -> ‘/var/www/fog//service/ipxe/bzImage’
‘packages/kernels/bzImage32’ -> ‘/var/www/fog//service/ipxe/bzImage32’
‘packages/inits/init.xz’ -> ‘/var/www/fog//service/ipxe/init.xz’
‘packages/inits/init_32.xz’ -> ‘/var/www/fog//service/ipxe/init_32.xz’
‘packages/clientfiles/FOGService.msi’ -> ‘/var/www/fog//client/FOGService.msi’
‘packages/clientfiles/SmartInstaller.exe’ -> ‘/var/www/fog//client/SmartInstaller.exe’
mysqlnd
Synchronizing state of apache2.service with SysV init with /lib/systemd/systemd-sysv-install…
Executing /lib/systemd/systemd-sysv-install enable apache2
Synchronizing state of php7.1-fpm.service with SysV init with /lib/systemd/systemd-sysv-install…
Executing /lib/systemd/systemd-sysv-install enable php7.1-fpm
Signature ok
subject=CN = 10.6.210.155
Getting CA Private Key
CA certificate and CA private key do not match
139662081185536:error:06067099:digital envelope routines:EVP_PKEY_copy_parameters:different parameters:…/crypto/evp/p_lib.c:93:
139662081185536:error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:…/crypto/x509/x509_cmp.c:294:
root@usktfoglp001:~/fogproject/bin/error_logs#

Sebastian Roth

@JYost said in Fog 1.5.5 to 1.5.8 - Creating SSL Certificate - FAILED:

CA certificate and CA private key do not match

Have you messed with the certificate authority (CA) files at some point? Maybe tried to use a custom CA?

Please run the following commands and post output here:

ls -alR /opt/fog/snapins/ssl/
ls -alR /var/www/html/fog/management/other/
md5sum /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/html/fog/management/other/ca.cert.pem
openssl x509 -noout -modulus -in /opt/fog/snapins/ssl/CA/.fogCA.pem | openssl md5
openssl rsa -noout -modulus -in /opt/fog/snapins/ssl/CA/.fogCA.key | openssl md5
openssl verify -verbose -CAfile /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/fog/management/other/ssl/srvpublic.crt

We can surely fix this but I shall ask you about the amount of host’s with fog-client installation before we can go ahead?! If you have many fog-clients installed we really want to be careful and try to fix this instead of re-creating the CA that would break all the fog-client installs.

JYost

On this particular FOG server (located in our Kent, Washinton office) we have approximately 125 clients on it. And no, I’ve done nothing with the CA. I have a number of other FOG servers around the world which are exhibiting identical behavior when the upgrade is attempted (Creating SSL Certificate Failed). All are running Ubuntu 16.04. FOG versions are all either 1.5.4. or 1.5.5. Should I go ahead and run the commands you gave me? It was unclear as to whether you wanted to hold off until you knew the number of clients we are dealing with on this server.

JYost

ls -alR /opt/fog/snapins/ssl OUTPUT:

fogadmin@usktfoglp001:/opt/fog/snapins$ ls -alR
.:
total 406536
drwxrwxrwx 3 fog      www-data      4096 Dec 10 06:23 .
drwxr-xr-x 5 root     root          4096 Jan  9  2019 ..
drwxrwxrwx 3 fog      www-data      4096 Apr  7 07:41 ssl
-rwxr-xr-x 1 fogadmin fogadmin 416273072 Dec 10 06:27 SymRedistributable.exe

./ssl:
total 24
drwxrwxrwx 3 fog      www-data 4096 Apr  7 07:41 .
drwxrwxrwx 3 fog      www-data 4096 Dec 10 06:23 ..
drwxrwxrwx 2 fog      www-data 4096 Dec 10 06:23 CA
-rw-r--r-- 1 root     root      102 Apr  7 08:41 ca.cnf
-rwxr-xr-x 1 fogadmin fogadmin 1590 Dec 10 06:23 fog.csr
-rwxrwxrwx 1 fog      www-data 3243 Jan  9  2019 .srvprivate.key

./ssl/CA:
total 24
drwxrwxrwx 2 fog      www-data 4096 Dec 10 06:23 .
drwxrwxrwx 3 fog      www-data 4096 Apr  7 07:41 ..
-rwxrwxrwx 1 fog      www-data 3243 Jan  9  2019 .fogCA.key
-rwxr-xr-x 1 fogadmin fogadmin 1801 Dec 10 06:23 .fogCA.pem
-rwxrwxrwx 1 fog      www-data   41 Apr  7 08:41 .fogCA.srl
-rwxr-xr-x 1 fogadmin fogadmin   17 Dec 10 06:23 .srl
fogadmin@usktfoglp001:/opt/fog/snapins$

ls -alR /var/www/html/fog/managment/other OUTPUT:

fogadmin@usktfoglp001:/var/www/html/fog/management/other$ ls -alR
.:
total 56
drwxr-xr-x  3 root root  4096 Apr  7 08:41 .
drwxr-xr-x 10 root root  4096 Apr  7 08:41 ..
-rw-r--r--  1 root root 35141 Apr  7 08:41 gpl-3.0.txt
-rw-r--r--  1 root root  6152 Apr  7 08:41 index.php
drwxr-xr-x  2 root root  4096 Apr  7 08:41 ssl

./ssl:
total 8
drwxr-xr-x 2 root root 4096 Apr  7 08:41 .
drwxr-xr-x 3 root root 4096 Apr  7 08:41 ..
-rw-r--r-- 1 root root    0 Apr  7 08:41 srvpublic.crt
fogadmin@usktfoglp001:/var/www/html/fog/management/other$

md5sum /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/html/fog/management/other/ca.cert.pm OUTPUT:

a4ee8c62634fc51e11a59965f8ebde85  /opt/fog/snapins/ssl/CA/.fogCA.pem
md5sum: /var/www/html/fog/management/other/ca.cert.pm: No such file or directory
fogadmin@usktfoglp001:~$

openssl x509 -noout -modulus -in /opt/fog/snapins/ssl/CA/.fogCA.pem | openssl md5 OUTPUT:

(stdin)= f8b599032ea17eddacb8cd45b94d32a5

openssl rsa -noout -modulus -in /opt/fog/snapins/ssl/CA/.fogCA.key | openssl md5 OUTPUT:

(stdin)= 707d08d4139a901ed6fec151abd68bf7

openssl verify -verbose -CAfile /opt/fog/snapins/ssl/CA/.fogCA.pem /var/www/fog/management/other/ssl/srvpublic.crt OUTPUT:

unable to load certificate
139785451083520:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE

JYost

Any thoughts as to my next course of action?

Sebastian Roth

@JYost Seems like something went wrong on 10th of Dezember 2019. Do you remember what was done back then?

I will try to replicate the issue in a test VM tomorrow but I don’t think we have much chance to fix it in the state it is now with one file being zero bytes and the backup copy of the CA cert missing.

Do you have a backup of the whole server from before 10th of Dez 2019? If not, can you re-deploy the fog-client via GPO or other ways? It is not working right now anyway.

JYost

Yes, we can re-deploy the fog client without any problem to all of our workstations. I don’t recall anything happening back on 12-19-19. We have had problems on all of our FOG servers since I took over this position in June of 2019 though. When the various sites image laptops/PC’s about 50% of the time the device will not rename properly or join the domain. We end up having to do that manually. Not sure if that problem is related to our Certificate issue - thoughts?

Sebastian Roth

@JYost Ok, lets tackle this. Make a backup copy of those directories, just to be sure:

sudo su -
tar cvzpf /root/backup_opt_fog_snapins.tar.gz /opt/fog/snapins
tar cvzpf /root/backup_var_www_html_fog_management_otherl.tar.gz /var/www/html/fog/management/other

Don’t worry about the warning on “Removing leading ‘/’ …”. Now go back to your fogproject directory and re-run the installer (as root) using command line switches:

cd /root/fogproject/bin
./installfog.sh --recreate-CA --recreate-keys

For other users reading this. Be aware that this will disconnect any fog-client installation you have from your FOG server!

When the various sites image laptops/PC’s about 50% of the time the device will not rename properly or join the domain.

It’s pretty much impossible to diagnose without having the fog.log file that you find on each of the hosts. A few things I can tell you nevertheless. Renaming is usually done as a last step of the deployment by modifying registry keys even before the host boots for the first time (if FOG Configuration -> FOG Settings -> General Settings -> CHANGE HOSTNAME EARLY is enabled). While I have not tested this feature lately we don’t have other reports of renaming to fail. So this part I wonder about.

Now the second part of renaming and domain join is done by the fog-client when the host boots up. Early host rename is not a pre-condition. The fog-client will just check if the host is named properly already and do the rename if not. But this depends on the fog-client actually being able to talk to the FOG server. The certificate mess we saw on this particular server would make it impossible for the fog-client to connect and do it’s job. So renaming and domain join (done by the fog-client) won’t work here. This should be fixed after the installer re-created the CA and you re-deployed the fog-client to all the hosts.

We have updated the fog-client with 1.5.8 (current is version 0.11.9) and you will probably want to use the latest version when pushing it out to the clients using the MSI switches for silent deploy.

JYost

I have performed the backup and re-installation of FOG as you directed and this time it appears to have completed without any errors. It still came up with the message that the current fogstorage database didn’t meet security standards…setup changed them for me, then gave me the credentials to copy down. It then said I will need to run the installer again after I change the .fogsettings file. But after I hit OK it did finish the install, and I do have access to the web interface.

1.) Do I need to change the snmysqluser and snmysqlpass values in the .fogsettings file?

2.) Do I need to run the installer again?

3.) Since I have several other servers displaying exactly the same (Creating SSL Certificate FAILED) error during the upgrade should I use the process you’ve outlined below to remedy those devices as well?

I will download a copy of the latest FOG client from the server and start deploying it to the workstations. I assume I will need to do this from each server which had the SSL error after I reinstall FOG, as each has it’s own IP internally configured in it?

Thank you VERY much for your help!!! The tech support on here is top notch and it’s awesome that you take the time to hand-hold new Linux users like myself

Sebastian Roth

@JYost said in Fog 1.5.5 to 1.5.8 - Creating SSL Certificate - FAILED:

1.) Do I need to change the snmysqluser and snmysqlpass values in the .fogsettings file?

The message displayed might be confusing as a FOG server is a storage node in itself. So this part of the message: “… you need to manually update on all your storage nodes …” actually means “on all storage nodes except this one”. Please don’t change these on your FOG master node!

2.) Do I need to run the installer again?

No, not one this FOG server but only on external FOG storage nodes if you have those.

3.) Since I have several other servers displaying exactly the same (Creating SSL Certificate FAILED) error during the upgrade should I use the process you’ve outlined below to remedy those devices as well?

For now I would suggest you re-deploy the fog-client for the hosts in this network and see how things go. Check the fog.log file on a couple of the hosts to make sure they do connect to the FOG server properly. If they do and everything seems to work you can go ahead and fix the others as well.

I will download a copy of the latest FOG client from the server and start deploying it to the workstations. I assume I will need to do this from each server which had the SSL error after I reinstall FOG, as each has it’s own IP internally configured in it?

The fog-client is not bound to the FOG server it was downloaded from. When installing the client you will need to specify the FOG server IP it should talk to and it will download the CA certificate from that particular FOG server and only talk to this server from then on. Therefore I posted a link to the wiki where MSI parameters are described to do a silent install of the fog-client pointing to a specific FOG server IP.

If you run into further issues with the fog-client I may ask you to open another topic for this particular issue. We try to keep topics sorted so other users find useful answers as well. Marking as solved as the initial issue is fixed.

JYost

Good morning Sebastian, one last question if I may…
Since all of my images were built with an old FOG client from version 1.5.0 and we just updated to 1.5.8 as well as reinstalled the Certificates - will I need to rebuild my images with the new FOG client from 1.5.8?

Thank you!

Sebastian Roth

@JYost The images themselves have not much to do with the fog-client software. It just does the host management (reboot on imaging tasks, domain join and so on).

But I do understand your question! Between 1.5.0 and 1.5.8 there were many changes in FOG and FOS (FOG Linux OS) which does all the actual work when imaging. We moved to newer versions with many tools in FOS, especially also partclone. But as far as our testing shows you should be able to use your old images without any change. Give it a try, just do a deployment to one of your hosts and see if it works. It should.

Fog 1.5.5 to 1.5.8 - Creating SSL Certificate - FAILED

190

11.7k

17.1k

154.7k