Apache2 for fog vhost - failed - Trying to update Trunk

RipAU · Apr 20, 2016, 11:25 PM

Hi, Guys.

I have a working fog SVN that hasn’t had any real issues but I decided to update it to the latest trunk version as I was going to try to create a storage node later on.
BUT when updating I’m finding that it is failing to restart apache2 with the error in the installer.

Downloading inits, kernels, and the fog client..............OK
 * Comparing checksums of kernels and inits....................Done
 * Enabling apache and fpm services on boot....................OK
 * Creating SSL Certificate....................................OK
 * Creating auth pub key and cert..............................OK
 * Resetting SSL Permissions...................................OK
 * Setting up SSL FOG Server...................................OK
 * Restarting Apache2 for fog vhost............................Failed!

This is the error from the error_logs

New password: Retype new password: Changing password for user fog.
passwd: all authentication tokens updated successfully.
Failed to execute operation: No such file or directory
â mariadb.service - MariaDB 10.0 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-04-21 08:34:58 ACST; 2s ago
  Process: 3578 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
  Process: 3811 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
  Process: 3668 ExecStartPost=/usr/libexec/mysql-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
  Process: 3633 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)
  Process: 3611 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 3667 (mysqld_safe)
   CGroup: /system.slice/mariadb.service
           ââ3667 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
           ââ3779 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock

Apr 21 08:34:57 fog2 systemd[1]: Starting MariaDB 10.0 database server...
Apr 21 08:34:57 fog2 mysqld_safe[3667]: 160421 08:34:57 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Apr 21 08:34:57 fog2 mysqld_safe[3667]: 160421 08:34:57 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Apr 21 08:34:58 fog2 systemd[1]: Started MariaDB 10.0 database server.
Signature ok
subject=/CN=10.254.14.77
Getting CA Private Key
ln: failed to create symbolic link â/var/www/html/fog/fogâ: File exists
Job for httpd.service failed. See "systemctl status httpd.service" and "journalctl -xe" for details.
â httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2016-04-21 08:43:29 ACST; 2s ago
  Process: 3964 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 3964 (code=exited, status=1/FAILURE)

Apr 21 08:43:28 fog2 systemd[1]: Starting The Apache HTTP Server...
Apr 21 08:43:29 fog2 systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Apr 21 08:43:29 fog2 systemd[1]: Failed to start The Apache HTTP Server.
Apr 21 08:43:29 fog2 systemd[1]: Unit httpd.service entered failed state.
Apr 21 08:43:29 fog2 systemd[1]: httpd.service failed.

â php-fpm.service - The PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-04-21 08:43:30 ACST; 2s ago
 Main PID: 3965 (php-fpm)
   Status: "Ready to handle connections"
   CGroup: /system.slice/php-fpm.service
           ââ3965 php-fpm: master process (/etc/php-fpm.conf
           ââ3970 php-fpm: pool www
           ââ3971 php-fpm: pool www
           ââ3972 php-fpm: pool www
           ââ3973 php-fpm: pool www
           ââ3974 php-fpm: pool www

Apr 21 08:43:28 fog2 systemd[1]: Starting The PHP FastCGI Process Manager...
Apr 21 08:43:30 fog2 systemd[1]: Started The PHP FastCGI Process Manager.

From httpd errors

[Thu Apr 21 08:52:48.951978 2016] [ssl:emerg] [pid 4062] AH02565: Certificate and private key 10.254.14.77:443:0 from /var/www/html/fog/management/other/ssl/srvpublic.crt and /opt/fog/snapins/ssl/.srvprivate.key do not match
AH00016: Configuration Failed

So would this be an issue with the .fogsettings file? I’m not sure what I’m missing.
Obviously there is an issue with the Certs but I’m not sure if it is something I have/haven’t done.

This server has a snapshot so I can easily go back to a working version just hoping to work out what I’m doing wrong with the update?

Thanks guys.

RipAU · Apr 20, 2016, 11:27 PM

Oh I forgot to mention this is a Redhad based server on Fedora 22.

Thanks again.

Wayne Workman · Apr 21, 2016, 2:16 AM

I don’t think anything in the .fogsettings file is causing this, but feel free to post the file so we can look it over - feel free to change the passwords in it to something like PASSWORDwasHERE

If you look at the timestamps on the logs from the installer, and the httpd_error log file, they don’t match by over 9 minutes. So I doubt that the httpd error you found is related - although it’s still concerning in and of itself.

Here’s the line that perked my interest:

Job for httpd.service failed. See "systemctl status httpd.service" and "journalctl -xe" for details.

What I’d do next is re-run the installer, and immediately after it fails, I’d issue the command journalctl -xe and see what’s in there. Please do share what you find.

RipAU · Apr 21, 2016, 3:16 AM

I think that was from me trying to do a manual restart of the httpd to see what the error was before I looked at the logs.

## Created by the FOG Installer
## Version: 6207
## Install time: Thu 17 Dec 2015 10:20:16 ACST

ipaddress="10.254.14.77";
interface="eno16780032";
submask="255.255.240.0";
routeraddress="         option routers      10.254.15.100;";
plainrouter="10.254.15.100";
dnsaddress="    option domain-name-servers      10.254.14.55; ";
dnsbootimage="10.254.14.55";
password='PASSWORDXX';
osid="1";
osname="Redhat";
dodhcp="N";
bldhcp="0";
blexports="1";
installtype="N";
snmysqluser=""
snmysqlpass='';
snmysqlhost="";
installlang="0";
donate="0";
storageLocation="/images";
fogupdateloaded="1";
storageftpuser="fog";
storageftppass='PASSWORDXX';
docroot="/var/www/html/";
webroot="fog/";
caCreated="yes";
startrange="10.254.14.10";
endrange="10.254.14.254";
bootfilename="undionly.kpxe";
packages=" httpd php php-cli php-common php-gd mysql mysql-server tftp-server nfs-utils vsftpd net-tools wget xinetd tar gzip make m4 gcc gcc-c++ lftp php-mysqlnd curl php-mcrypt php-mbstring mod_ssl php-fpm php-process";

noTftpBuild=''
notpxedefaultfile=''

Cheers.

Wayne Workman · Apr 20, 2016, 9:46 PM

@RipAU Your settings file looks fine - it’s in the older style right now because you have yet to complete the installation with a newer fog trunk copy - it’s fine.

Do you have a certificate/cert installed in Apache for use with the FOG web interface (to enable https)?

RipAU · Apr 21, 2016, 3:58 AM

Yeah it does that a default one from the initial fedora install, but in the Error logs it is pointing at the fog directory as the cert error?

I just tried the install again and this is from the fog server right after the install fails to restart httpd and the error in the logs until i did the install.

[Thu Apr 21 13:16:06.064600 2016] [ssl:emerg] [pid 15945] AH02565: Certificate and private key 10.254.14.77:443:0 from /var/www/html/fog/management/other/ssl/srvpublic.crt and /opt/fog/snapins/ssl/.srvprivate.key do not match
AH00016: Configuration Failed

This is also from the journalctl -xe. I haven’t really used the journalctl so I’m not that familiar with it.

-- The start-up result is done.
Apr 21 13:07:37 fog2 audit[1]: <audit-1130> pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mariadb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 21 13:07:37 fog2 polkitd[628]: Unregistered Authentication Agent for unix-process:15587:1584643 (system bus name :1.37, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8) (disconnec
Apr 21 13:07:40 fog2 polkitd[628]: Registered Authentication Agent for unix-process:15832:1585006 (system bus name :1.40 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/Authen
Apr 21 13:07:40 fog2 systemd[1]: Stopping The Apache HTTP Server...
-- Subject: Unit httpd.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has begun shutting down.
Apr 21 13:07:40 fog2 systemd[1]: Stopping The PHP FastCGI Process Manager...
-- Subject: Unit php-fpm.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit php-fpm.service has begun shutting down.
Apr 21 13:07:40 fog2 systemd[1]: Stopped The PHP FastCGI Process Manager.
-- Subject: Unit php-fpm.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit php-fpm.service has finished shutting down.
Apr 21 13:07:40 fog2 audit[1]: <audit-1130> pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=php-fpm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 21 13:07:40 fog2 audit[1]: <audit-1131> pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=php-fpm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 21 13:07:41 fog2 systemd[1]: Stopped The Apache HTTP Server.
-- Subject: Unit httpd.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has finished shutting down.
Apr 21 13:07:41 fog2 audit[1]: <audit-1131> pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 21 13:07:41 fog2 polkitd[628]: Unregistered Authentication Agent for unix-process:15832:1585006 (system bus name :1.40, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8) (disconnec
Apr 21 13:15:59 fog2 polkitd[628]: Registered Authentication Agent for unix-process:15897:1634901 (system bus name :1.41 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/Authen
Apr 21 13:16:00 fog2 systemd[1]: Reloading.
Apr 21 13:16:01 fog2 systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceedi
Apr 21 13:16:02 fog2 polkitd[628]: Unregistered Authentication Agent for unix-process:15897:1634901 (system bus name :1.41, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8) (disconnec
Apr 21 13:16:02 fog2 polkitd[628]: Registered Authentication Agent for unix-process:15934:1635272 (system bus name :1.42 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/Authen
Apr 21 13:16:02 fog2 systemd[1]: Stopped The Apache HTTP Server.
-- Subject: Unit httpd.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has finished shutting down.
Apr 21 13:16:02 fog2 systemd[1]: Stopped The PHP FastCGI Process Manager.
-- Subject: Unit php-fpm.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit php-fpm.service has finished shutting down.
Apr 21 13:16:02 fog2 polkitd[628]: Unregistered Authentication Agent for unix-process:15934:1635272 (system bus name :1.42, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8) (disconnec
Apr 21 13:16:04 fog2 polkitd[628]: Registered Authentication Agent for unix-process:15940:1635487 (system bus name :1.43 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/Authen
Apr 21 13:16:04 fog2 systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has begun starting up.
Apr 21 13:16:05 fog2 systemd[1]: Starting The PHP FastCGI Process Manager...
-- Subject: Unit php-fpm.service has begun start-up
Apr 21 13:16:06 fog2 systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Apr 21 13:16:06 fog2 systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has failed.
--
-- The result is failed.
Apr 21 13:16:06 fog2 systemd[1]: Unit httpd.service entered failed state.
Apr 21 13:16:06 fog2 systemd[1]: httpd.service failed.
Apr 21 13:16:06 fog2 audit[1]: <audit-1130> pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Apr 21 13:16:06 fog2 systemd[1]: Started The PHP FastCGI Process Manager.
-- Subject: Unit php-fpm.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit php-fpm.service has finished starting up.
--
-- The start-up result is done.
Apr 21 13:16:06 fog2 audit[1]: <audit-1130> pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=php-fpm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 21 13:16:06 fog2 polkitd[628]: Unregistered Authentication Agent for unix-process:15940:1635487 (system bus name :1.43, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8) (disconnec
lines 2911-2972/2972 (END)

So I’m leaning towards a cert error with the fog sever? I’m assuming it is regenerating some certs that are then failing to match?
I’m not that familiar with the SSL system in Linux sorry.

RipAU · Apr 21, 2016, 4:04 AM

Also my httpd fog.conf file

<VirtualHost *:80>
    KeepAlive Off
    ServerName 10.254.14.77
    DocumentRoot /var/www/html/
    #RewriteEngine On
    #RewriteRule /management/other/ca.cert.der$ - [L]
    #RewriteRule /management/ https://%{HTTP_HOST}%{REQUEST_URI}%{QUERY_STRING} [R,L]
</VirtualHost>
<VirtualHost *:443>
    KeepAlive Off
    Servername 10.254.14.77
    DocumentRoot /var/www/html/
    SSLEngine On
    SSLCertificateFile /var/www/html/fog//management/other/ssl/srvpublic.crt
    SSLCertificateKeyFile /opt/fog/snapins/ssl/.srvprivate.key
    SSLCertificateChainFile /var/www/html/fog//management/other/ca.cert.der
</VirtualHost>

All looks pretty fine to me. These are the files that after the update the server complains about though… ?

Wayne Workman · Apr 21, 2016, 4:09 AM

@RipAU said in Apache2 for fog vhost - failed - Trying to update Trunk:

These are the files that after the update the server complains about though… ?

Yes, they are. Here a while back, the locations of these files were changed. Used to be it was /opt/fog/.ssl I think?? And now it’s /opt/fog/snapins/ssl and I’m wondering if something went wrong with the moving around.

Do you know what version of fog trunk you came from? I’d like to try to replicate if I can.

RipAU · Apr 21, 2016, 4:16 AM

Going from the Fog installer log.

## Created by the FOG Installer
## Version: 6207
## Install time: Thu 17 Dec 2015 10:20:16 ACST

I’m assuming it updates the log at the last update?

RipAU · Apr 21, 2016, 7:00 AM

@Wayne-Workman said in Apache2 for fog vhost - failed - Trying to update Trunk:

Yes, they are. Here a while back, the locations of these files were changed. Used to be it was /opt/fog/.ssl I think?? And now it’s /opt/fog/snapins/ssl

I’m just checking now as well, I don’t have any files in /opt/fog/.ssl I only have the following structure:

/opt/fog/log
/opt/fog/service
/opt/fog/snapins
/opt/fog/utils
/opt/fog/.fogsettings

So I’m suspecting that something is going wrong with regenerating the keys from the CA.

RipAU · Apr 21, 2016, 11:09 PM

After going through the forum some more I found this - ./installfog.sh --recreate-keys
Would this help fix the issue at all with the upgrade?

https://forums.fogproject.org/topic/6975/change-server-but-not-change-certificate/6

From reading this I should be able to run this without affecting the currently running clients? as it should use the current CA?

Wayne Workman · Apr 21, 2016, 11:41 PM

@RipAU said in Apache2 for fog vhost - failed - Trying to update Trunk:

Would this help fix the issue at all with the upgrade?
https://forums.fogproject.org/topic/6975/change-server-but-not-change-certificate/6
From reading this I should be able to run this without affecting the currently running clients? as it should use the current CA?

Wouldn’t hurt to try, not at all. Here’s further reading on the topic for yourself and future readers: https://wiki.fogproject.org/wiki/index.php?title=FOG_Client

RipAU · Apr 22, 2016, 2:27 AM

Ok this worked by recreating the keys. Install worked fine and server seems to be working well.
Only thing I have noticed is several labs worth of Desktops had invalid tokens so I’ve needed to “Reset Encryption Data” to get them talking back to the server again.

Aside from that all is working well

Thanks again.

Apache2 for fog vhost - failed - Trying to update Trunk

160

12.0k

17.3k

155.2k