FOG forcing schema update post image upload

[Kiweegie]

Running trunk@:4542 version of FOG and had been running well. I uploaded a new (pre-sysprep) image for Windows 7 successfully this morning. Ran unattend.xml on the reference pc then ran image upload for a postsysprep image (this is on same box). That failed but due to syntax errors in the unatend.xml file however weirdly when I tried to access FOG server to redeploy the presysprep image back to that machine i was presented with the schema update page.

Clicking the install/upgrade button takes me to an error page, beginning of which shows as follows. I’ve uploaded complete error log as text attachment.

0_1452094345876_schema_update_error_06Jan16.txt

***Install/Upgrade Successful!

The following errors occured

Update ID: 1 - 0

Database Error:

Connection refused, Message: Check that database is running

Database SQL:

CREATE DATABASE fog

Update ID: 1 - 1

Database Error:

Connection refused, Message: Check that database is running

Database SQL:

CREATE TABLE fog.groupMembers ( gmID int(11) NOT NULL auto_increment, gmHostID int(11) NOT NULL, gmGroupID int(11) NOT NULL, PRIMARY KEY (gmID),
KEY new_index (gmHostID),
KEY new_index1 (gmGroupID) ) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC***

I tried running installer again as a “quick fix” but fails at:

 * Setting up and starting MySQL...............................Failed!

I can’t see anything obvious in /var/log/mysqld.log either.

Anyone else seen this or know where I can look to check for clues?

cheers Kiweegie.

[Wayne Workman]

If you moved to the latest version, fog now redirects errors to /var/log/fog_error_[version].log

[Kiweegie]

@Wayne-Workman Thanks Wayne, checked /var/log/fog… and there were logs for replication etc but no MySQL logs.

I was in a hurry so flattened the VM and built afresh however now faced with a new issue, similar to one I reported on a storage node on this post albeit this time its a Normal Node.

Pulled latest git this evening 2000 GMT (git log shows trunk@4603)

And install fails at this point:

 * Installing package: gcc.....................................OK
 * Installing package: gcc-c++.................................OK
 * Installing package: lftp....................................OK
 * Installing package: php-mysqlnd.............................Failed!

This is on a fresh minimal install of CentOS 6.7 with all updates and nothing other than git installed.

I recalled that on the other call (or possibly another, sorry its late here and its been a long fraught day…) you mentioned about installing twice. May also have been someone else completely

So I tried installing again and this time got to the /images selection part of the install but failed here:

 * What is the storage location for your images directory? (/images)
 * Setting up fog user.........................................OK
OK
 * Setting up and starting MySQL...............................OK
 * Backing up user reports.....................................OK
 * Stopping web service........................................Failed!

What the heck, a third install attempt, this time failed same place as above but this time also added in a fog username and password

* New password set for:
                username: fog
                password: LlmqulgCso1w5i0IjN4L/O3HjsJXU6kNDC94QElxopY=

Also while I think of it been meaning to ask this for a while, is there a reason during the DHCP portion of the installer that it includes the (eth*) at the prompt? e.g.

What is the IP address to be used for the router on
      the DHCP server? [eth0]10.***.******

The very first time I saw this I was a little nonplussed as there was no obvious “prompt”. All other portions of the installer where they include options in brackets allow a simple enter key tap to accept the default entry within the brackets for example

What type of installation would you like to do? [N/s (Normal/Storage)]

New users may initially be thrown at this point is my only reason for raising this.

Having typed all the above wondering if better to log on 2 separate new calls - let me know if thats preferred.

cheers, Kiweegie.

[Wayne Workman]

@Kiweegie Again, we need to see the error output.

Execute this command: ls /var/log

You should see these:

The version you just tried installing is the one that has the errors we need to see.

[Kiweegie]

@Wayne-Workman Just to rule out issues with my work environment I’ve spun up two local VMs at home one on CentOS 6.7, one on Cent0S 7.

The Centos 6 failed in same manner as my work machine described earlier.

CentOS 7 machine I noticed (to digress briefly again) that the DHCP part of the installer picks up the DHCP address (I’ve used DHCP instead of statics on tests so that may be reason, though CentOS 6 did not so this is more likely a 6 vs 7 thing)

  Would you like to setup a router address for the DHCP server? [Y/n]
  What is the IP address to be used for the router on
      the DHCP server? [192.168.245.2]

However above is a moot point as CentOS 7 failed with same error

* Setting up and starting MySQL...............................Failed!

Again 2nd installation attempt displays fog user and password but subsequently fails with same error as above.

regards Kiweegie.

[Kiweegie]

@Wayne-Workman HI Wayne, must have been having a senior moment. Here’s what you’re after.

/usr/bin/lsb_release
grep: /opt/fog/.fogsettings: No such file or directory
../lib/common/functions.sh: line 397: epel-release: command not found
curl: (22) The requested URL returned error: 404 Not Found
error: skipping http://rpms.famillecollet.com/enterprise/remi-release-6.7.rpm - transfer failed
Retrieving http://rpms.famillecollet.com/enterprise/remi-release-6.7.rpm
New password: Retype new password: Changing password for user fog.
passwd: all authentication tokens updated successfully.
Stopping mysqld:  ESC[60G[ESC[0;32m  OK  ESC[0;39m]
Starting mysqld:  ESC[60G[ESC[0;32m  OK  ESC[0;39m]
mysqld (pid  15732) is running...
Stopping httpd: ESC[60G[ESC[0;31mFAILEDESC[0;39m]
php-fpm: unrecognized service

So its falling over at the php-fpm piece again…

I just tried installing php-fpm again manually and then running FOG installer again

That gets a little further but fails here

 * Setting up and starting MySQL...............................OK
 * Backing up user reports.....................................OK
 * Stopping web service........................................OK
 * Is the MySQL password blank? (Y/n)
 * Setting up Apache and PHP files.............................Failed!

Checking log file again I get this

/usr/bin/lsb_release
grep: /opt/fog/.fogsettings: No such file or directory
../lib/common/functions.sh: line 397: epel-release: command not found
curl: (22) The requested URL returned error: 404 Not Found
error: skipping http://rpms.famillecollet.com/enterprise/remi-release-6.7.rpm - transfer failed
Retrieving http://rpms.famillecollet.com/enterprise/remi-release-6.7.rpm
New password: Retype new password: Changing password for user fog.
passwd: all authentication tokens updated successfully.
Stopping mysqld:  ESC[60G[ESC[0;32m  OK  ESC[0;39m]
Starting mysqld:  ESC[60G[ESC[0;32m  OK  ESC[0;39m]
mysqld (pid  16225) is running...
Stopping httpd: ESC[60G[ESC[0;31mFAILEDESC[0;39m]
Stopping php-fpm: ESC[60G[ESC[0;31mFAILEDESC[0;39m]
sed: no input files
sed: no input files

Not sure where to go from there.

cheers Kiweegie.

[Kiweegie]

@Tom-Elliott Hi Tom, thanks for you efforts last night sorting this issue for me over teamviewer. I have been configuring the FOG server today, re importing hosts insstalling plugins and generally getting it back to where it was. I’ve encountered similar error/issue as before where the FOG server database decided to screw up.

Initially was having problems logging in - checked status of mysqld and httpd both showed up ok. Bounced the box and on restart same issues - checked status and httpd had not started. Started manually and did chkconfig httpd on tried restarting again to check httpd works - both httpd and mysqld running.

However. Isn’t there always one of them? Or a but? On launching Web GUI I’m once more presented with the Schema update page

I’ve tried clicking the Export-Backujp DB button but nothing happens at all - opens URL http://My_Fog_Server/fog/management/export.php?node=schemaupdater and then hangs.

Quick check of the MySQL docs shows --allowkeywords should actually be --allow-keywords - guess the button just calls the code written, perhaps needs an update.

I’ve attached output of /var/log/fog_error_5866.log

0_1452181730183_fog_error_5866.log

Trying a service mysqld restart stops the server ok but now fails to start it so something is royally %(&^ here. Oddly on restarting the server completely mysqld starts but httpd is stopped. Starting manually now throws error.

[root@my_fog_server ~]# service mysqld status
mysqld (pid 3225) is running...
[root@my_fog_server ~]# service httpd status
httpd is stopped
[root@my_fog_server ~]# service httpd start
Starting httpd: [Thu Jan 07 15:55:28 2016] [warn] _default_ VirtualHost overlap on port 443, the first has precedence
                                                           [  OK  ]

/etc/httpd/conf.d/fog.conf shows

<VirtualHost *:80>
    KeepAlive Off
    ServerName My_FOG_IP
    DocumentRoot /var/www/html/
    #RewriteEngine On
    #RewriteRule /management/other/ca.cert.der$ - [L]
    #RewriteRule /management/ https://%{HTTP_HOST}%{REQUEST_URI}%{QUERY_STRING} [R,L]
</VirtualHost>
<VirtualHost *:443>
    KeepAlive Off
    Servername My_FOG_IP
    DocumentRoot /var/www/html/
    SSLEngine On
    SSLCertificateFile /var/www/html/fog//management/other/ssl/srvpublic.crt
    SSLCertificateKeyFile /opt/fog/snapins/ssl/.srvprivate.key
    SSLCertificateChainFile /var/www/html/fog//management/other/ca.cert.der
</VirtualHost>

There is also a <VirtualHost default:443> stanza in /etc/httpd/conf.d/ssl.conf

To remind you this was on a clean minimal install of CentOS 6.5 upgraded to 6.7 via yum update and with only git installed pre FOG installation. Its a virtual machine running on VSphere 6.0 in case that has any bearing on things.

cheers, Kiweegie.

[Wayne Workman]

@Kiweegie
inside of /etc/httpd/conf.d/fog.conf

SSLCertificateFile /var/www/html/fog//management/other/ssl/srvpublic.crt
SSLCertificateKeyFile /opt/fog/snapins/ssl/.srvprivate.key
SSLCertificateChainFile /var/www/html/fog//management/other/ca.cert.der

See the double forward slashes? Try to take them out and see what happens.

[Kiweegie]

@Wayne-Workman HI Wayne

while looking at the above I actually saw another alert on the box and have found one of the problems at least and the cause of the SQL crash etc - no blasted disk space!!!

This is on a VM with 150GB assigned but the OS for some completely unknown reason was showing as only 50GB and by the time I’d re-added all my original images the space was hosed.

I’ve nuked the VM and built from scratch to be on safe side and OS now sees the correct amount of space. Ran into a new hurdle with the very latest git version however during install.

No prompt during installation for normal/storage node, IP address selection, DHCP or DNS etc.

Just the following:

######################################################################
     #           ** Notice ** FOG is difficult to setup securely          #
     #        SELinux and IPTables are usually asked to be disabled       #
     #           There have been strides in adding capabilities           #
     #          The recommendations would now be more appropriate         #
     #    to set SELinux to permissive and to disable firewall for now.   #
     #  You can find some methods to enable SELinux and maintain firewall #
     #   settings and ports. If you feel comfortable doing so please do   #
     ######################################################################
     #            Please see our wiki for more information at:            #
     ######################################################################
     #             https://wiki.fogproject.org/wiki/index.php             #
     ######################################################################

                      Here are the settings FOG will use:
                                  Base Linux:
                      Detected Linux Distribution: CentOS
                              Server IP Address:
                                  Interface:


 * Are you sure you wish to continue (Y/N) y

                              Installation Started

                  Installing required packages, if this fails
               make sure you have an active internet connection.

 * Adding needed repository....................................OK
 * Preparing Package Manager...................................OK
 * Packages to be installed:




 * Updating packages as needed.................................OK

                        Confirming package installation.


                             Configuring services.


 * What is the storage location for your images directory? (/images)

so while I’ve resolved the initial cause of the schema update page showing up I need some assistance looking at latest installer please (trunk@4613)

cheers Kiweegie.

[Wayne Workman]

@Kiweegie Delete your /opt/fog/.fogsettings file and that should bring back the rest of the menu. Tom will fix it when he has time.

rm -f /opt/fog/.fogsettings

[Kiweegie]

Tom Elliot has resolved the installer not asking for any info issue but now fails with error:

* Adding needed repository....................................Failed!

I’ve checked the log at /var/log/fog_error_5893.log which shows

Package epel-release-6-8.noarch already installed and latest version
Nothing to do
package http://rpms.remirepo.net/enterprise/remi-release-6.7.rpm is not installed
Loaded plugins: fastestmirror
Setting up Install Process
Cannot open: http://rpms.remirepo.net/enterprise/remi-release-6.7.rpm. Skipping.
Error: Nothing to do
../lib/common/functions.sh: line 439: yum-config-manager: command not found

So the remi repo needed to be enabled, doing this alone was not sufficient to fix problem, Tom Elliot also had to enable yum-config-manager. Tom has updated this in git and i’m now running on trunk@4617

Tom/Wayne, thanks again for your efforts, really do appreciate it.

regards Kiweegie.