build 7102 'Backing up database ... failed'

sudburr

Updating to build 7102 from 6821 has worked on 5 other servers so far, but I have one that insists on failing at …

Backing up database.........................................Failed!

From /opt/trunksvn/fogproject/bin/error_logs

Apr 07 00:01:21 xyzfog systemd[1]: Starting The PHP FastCGI Process Manager...
Apr 07 00:01:21 xyzfog systemd[1]: Started The PHP FastCGI Process Manager.
--2016-04-07 00:01:23--  http://172.19.244.32/fog//management/export.php?type=sql
Connecting to 172.19.244.32:80... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.
... Giving up. after 20 attempts

Also /var/log/httpd/error_log:

[Wed Apr 06 23:32:53.046956 2016] [:error] [pid 2919] [client 192.168.54.120:57605] PHP Warning:  mysqli::real_connect(): MySQL server has gone away in /var/www/html/fog/lib/db/mysql.class.php on line 29
[Wed Apr 06 23:32:57.487734 2016] [:error] [pid 2852] [client 172.22.57.44:49402] PHP Warning:  mysqli::real_connect(): MySQL server has gone away in /var/www/html/fog/lib/db/mysql.class.php on line 29
[Wed Apr 06 23:32:57.487869 2016] [:error] [pid 2852] [client 172.22.57.44:49402] PHP Warning:  mysqli::real_connect(): (HY000/2002): Connection refused in /var/www/html/fog/lib/db/mysql.class.php on line 29
[Wed Apr 06 23:32:57.487883 2016] [:error] [pid 2852] [client 172.22.57.44:49402] PHP Warning:  mysqli::set_charset(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 30
[Wed Apr 06 23:32:57.487896 2016] [:error] [pid 2852] [client 172.22.57.44:49402] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 39
[Wed Apr 06 23:32:57.487915 2016] [:error] [pid 2852] [client 172.22.57.44:49402] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 39
[Wed Apr 06 23:32:57.487927 2016] [:error] [pid 2852] [client 172.22.57.44:49402] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 51

Each server is identically configured on CentOS7, although … The first 4 servers were originally installed onto CentOS7.1.1503, this one was originally built upon CentOS7.2.1511.

Sebastian Roth

@sudburr The error logs seem to point to the DB not being properly up!? Tried re-starting and connecting via console client yet?

sudo systemctl restart mysql.service
mysql --user=root -p
Enter password:

Maybe you need to take a look in the system log (/var/log/messages on CentOS I think) if restarting the mysql fails.

sudburr

I can actually visit the schema updater page and click through it. Even though the installer has bailed, the server appears to be working fine. …

sudburr

Visiting the server via another session, while it was stuck on backing up database, I tried:

systemctl restart mysql.service
systemctl restart httpd.service
systemctl restart httpd

No change.

So far everything appears to still be working. I can export hosts, images, settings. This particular server was a recent rebuild from fresh iron and doesn’t have any hosts yet so that export is 0 KB .

sudburr

I’m building a duplicate server right now from the same source files to see what shakes out.

sudburr

So far I’ve tried updating to 7102, 7104 and 7106 on this server and it continues to fail at Backing up database.

My fresh iron build using the same source files and installed identically, works. yargen.

A little help? What has broken on this naughty server?

george1421

Is your FOG server behind a proxy server? I have seen where the fog server is behind a proxy server and you either use the environ variables or update wgetrc with proxy settings. This will cause the backup script to fail.

sudburr

No proxy server. Again, 4/5 worked, but number 5 is not alive.

george1421

Hmmm…

I think what I would do on this is look into the /opt/fog/.fogsettings file. That will have the parameters saved by the installer and the password selected for the mysql database. Then try to login to mysql using the FOG SQL password.

with mysql -u fog fog -p
That reads use user fog to connect to the fog database and prompt me for the password.

See if you can get logged into the mysql database. If you get logged in then do something like show tables; to list all of the fog related tables.

[edit] wait this doesn’t work on my fog server. I can’t test it right now I have to run [/edit]

sudburr

Command results:

mysql -u root fog -p
show tables;

+------------------------+
| Tables_in_fog          |
+------------------------+
| aloLog                 |
| clientUpdates          |
| dirCleaner             |
| globalSettings         |
| greenFog               |
| groupMembers           |
| groups                 |
| history                |
| hostAutoLogOut         |
| hostMAC                |
| hostScreenSettings     |
| hosts                  |
| imageGroupAssoc        |
| imagePartitionTypes    |
| imageTypes             |
| images                 |
| imagingLog             |
| inventory              |
| ipxeTable              |
| keySequence            |
| moduleStatusByHost     |
| modules                |
| multicastSessions      |
| multicastSessionsAssoc |
| nfsFailures            |
| nfsGroupMembers        |
| nfsGroups              |
| os                     |
| oui                    |
| plugins                |
| printerAssoc           |
| printers               |
| pxeMenu                |
| scheduledTasks         |
| schemaVersion          |
| snapinAssoc            |
| snapinGroupAssoc       |
| snapinJobs             |
| snapinTasks            |
| snapins                |
| supportedOS            |
| taskLog                |
| taskStates             |
| taskTypes              |
| tasks                  |
| userCleanup            |
| userTracking           |
| users                  |
| virus                  |
+------------------------+
49 rows in set (0.00 sec)

Wayne Workman

@sudburr I’ve been watching this thread and I’ve not had any ideas. The database backups go into /home maybe make sure you have space there. df -h

sudburr

Gigs worth of space.

Wayne Workman

@sudburr partitions, man. partitions… check anyways.

sudburr

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       115G  3.3G  106G   4% /
devtmpfs        919M     0  919M   0% /dev
tmpfs           928M     0  928M   0% /dev/shm
tmpfs           928M  8.5M  920M   1% /run
tmpfs           928M     0  928M   0% /sys/fs/cgroup
/dev/sdb1       932G  249G  684G  27% /images
/dev/sda1       477M  172M  276M  39% /boot
tmpfs           186M     0  186M   0% /run/user/1000

george1421

@Wayne-Workman what happens if the destination location does not exist?

Wayne Workman

@george1421 Not sure… I’m sure it exists though.

ls /home

Wayne Workman

r7116 on Fedora 21 fully updated, I’m getting a db failed message as well.

* Backing up database.........................................Failed!

Here’s what’s in the error log:

/usr/bin/lsb_release
/usr/bin/systemctl
ln: failed to create symbolic link ‘/usr/lib/systemd/system/mysql.service’: File exists
ln: failed to create symbolic link ‘/usr/lib/systemd/system/mysqld.service’: File exists
ln: failed to create symbolic link ‘/etc/systemd/system/mysql.service’: File exists
ln: failed to create symbolic link ‘/etc/systemd/system/mysqld.service’: File exists
/opt/fog/.fogsettings
Loaded plugins: langpacks
http://remi.kazukioishi.net/fedora/21/remi/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://remi.kazukioishi.net/fedora/21/remi/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
Trying other mirror.
No package epel-release available.
Error: Nothing to do
remi-release-21-3.fc21.remi.noarch
Loaded plugins: langpacks
================================== repo: remi ==================================
[remi]
async = True
bandwidth = 0
base_persistdir = /var/lib/yum/repos/x86_64/21
baseurl =
cache = 0
cachedir = /var/cache/yum/x86_64/21/remi
check_config_file_age = True
cost = 1000
deltarpm_metadata_percentage = 100
deltarpm_percentage =
enabled = True
enablegroups = True
exclude =
failovermethod = roundrobin
gpgcadir = /var/lib/yum/repos/x86_64/21/remi/gpgcadir
gpgcakey =
gpgcheck = True
gpgdir = /var/lib/yum/repos/x86_64/21/remi/gpgdir
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-remi
hdrdir = /var/cache/yum/x86_64/21/remi/headers
http_caching = all
includepkgs =
ip_resolve =
keepalive = True
keepcache = False
mddownloadpolicy = sqlite
mdpolicy = group:small
mediaid =
metadata_expire = 21600
metadata_expire_filter = read-only:present
metalink =
minrate = 0
mirrorlist = http://rpms.remirepo.net/fedora/21/remi/mirror
mirrorlist_expire = 86400
name = Remi's RPM repository - Fedora 21 - x86_64
old_base_cache_dir =
password =
persistdir = /var/lib/yum/repos/x86_64/21/remi
pkgdir = /var/cache/yum/x86_64/21/remi/packages
proxy = False
proxy_dict =
proxy_password =
proxy_username =
repo_gpgcheck = False
retries = 10
skip_if_unavailable = False
ssl_check_cert_permissions = True
sslcacert =
sslclientcert =
sslclientkey =
sslverify = True
throttle = 0
timeout = 30.0
ui_id = remi/21
ui_repoid_vars = releasever,
   basearch
username =

Loaded plugins: langpacks
httpd-2.4.16-1.fc21.x86_64
php-5.6.20-1.fc21.remi.x86_64
php-cli-5.6.20-1.fc21.remi.x86_64
php-common-5.6.20-1.fc21.remi.x86_64
php-gd-5.6.20-1.fc21.remi.x86_64
mariadb-10.0.21-1.fc21.x86_64
mariadb-server-10.0.21-1.fc21.x86_64
tftp-server-5.2-15.fc21.x86_64
nfs-utils-1.3.1-6.4.fc21.x86_64
vsftpd-3.0.2-12.fc21.x86_64
net-tools-2.0-0.31.20141124git.fc21.x86_64
wget-1.16.3-1.fc21.x86_64
xinetd-2.3.15-15.fc21.x86_64
tar-1.27.1-8.fc21.x86_64
gzip-1.6-8.fc21.x86_64
make-4.0-3.fc21.x86_64
m4-1.4.17-6.fc21.x86_64
gcc-4.9.2-6.fc21.x86_64
gcc-c++-4.9.2-6.fc21.x86_64
lftp-4.5.4-4.fc21.x86_64
Loaded plugins: langpacks
Installed Packages
php-mysqlnd.x86_64                  5.6.20-1.fc21.remi                   @remi
Available Packages
php-mysqlnd.i686                    5.6.15-1.fc21                        updates
php-mysqlnd-5.6.20-1.fc21.remi.x86_64
curl-7.37.0-14.fc21.x86_64
php-mcrypt-5.6.20-1.fc21.remi.x86_64
php-mbstring-5.6.20-1.fc21.remi.x86_64
mod_ssl-2.4.16-1.fc21.x86_64
php-fpm-5.6.20-1.fc21.remi.x86_64
php-process-5.6.20-1.fc21.remi.x86_64
Loaded plugins: langpacks
No packages marked for update
httpd-2.4.16-1.fc21.x86_64
php-5.6.20-1.fc21.remi.x86_64
php-cli-5.6.20-1.fc21.remi.x86_64
php-common-5.6.20-1.fc21.remi.x86_64
php-gd-5.6.20-1.fc21.remi.x86_64
mariadb-10.0.21-1.fc21.x86_64
mariadb-server-10.0.21-1.fc21.x86_64
tftp-server-5.2-15.fc21.x86_64
nfs-utils-1.3.1-6.4.fc21.x86_64
vsftpd-3.0.2-12.fc21.x86_64
net-tools-2.0-0.31.20141124git.fc21.x86_64
wget-1.16.3-1.fc21.x86_64
xinetd-2.3.15-15.fc21.x86_64
tar-1.27.1-8.fc21.x86_64
gzip-1.6-8.fc21.x86_64
make-4.0-3.fc21.x86_64
m4-1.4.17-6.fc21.x86_64
gcc-4.9.2-6.fc21.x86_64
gcc-c++-4.9.2-6.fc21.x86_64
lftp-4.5.4-4.fc21.x86_64
php-mysqlnd-5.6.20-1.fc21.remi.x86_64
php-mysqlnd-5.6.20-1.fc21.remi.x86_64
curl-7.37.0-14.fc21.x86_64
php-mcrypt-5.6.20-1.fc21.remi.x86_64
php-mbstring-5.6.20-1.fc21.remi.x86_64
mod_ssl-2.4.16-1.fc21.x86_64
php-fpm-5.6.20-1.fc21.remi.x86_64
php-process-5.6.20-1.fc21.remi.x86_64
New password: Retype new password: Changing password for user fog.
passwd: all authentication tokens updated successfully.
Failed to execute operation: No such file or directory
● mariadb.service - MariaDB 10.0 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled)
  Drop-In: /usr/lib/systemd/system/mariadb.service.d
           └─mysql-compat.conf
   Active: active (running) since Fri 2016-04-08 12:56:04 CDT; 2s ago
  Process: 47341 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
  Process: 47189 ExecStartPost=/usr/libexec/mysql-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
  Process: 47153 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)
  Process: 47129 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 47188 (mysqld_safe)
   CGroup: /system.slice/mariadb.service
           ├─47188 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
           └─47304 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock

Apr 08 12:56:03 localhost.localdomain mysqld_safe[47188]: 160408 12:56:03 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Apr 08 12:56:03 localhost.localdomain mysqld_safe[47188]: 160408 12:56:03 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Signature ok
subject=/CN=10.2.1.11
Getting CA Private Key
ln: failed to create symbolic link ‘/var/www/html/fog/fog’: File exists
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: active (running) since Fri 2016-04-08 13:05:04 CDT; 2s ago
 Main PID: 47564 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─47564 /usr/sbin/httpd -DFOREGROUND
           ├─47572 /usr/sbin/httpd -DFOREGROUND
           ├─47573 /usr/sbin/httpd -DFOREGROUND
           ├─47574 /usr/sbin/httpd -DFOREGROUND
           ├─47575 /usr/sbin/httpd -DFOREGROUND
           ├─47576 /usr/sbin/httpd -DFOREGROUND
           ├─47582 /usr/sbin/httpd -DFOREGROUND
           └─47585 /usr/sbin/httpd -DFOREGROUND

Apr 08 13:05:04 localhost.localdomain httpd[47564]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message

● php-fpm.service - The PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled)
   Active: active (running) since Fri 2016-04-08 13:05:04 CDT; 3s ago
 Main PID: 47565 (php-fpm)
   Status: "Ready to handle connections"
   CGroup: /system.slice/php-fpm.service
           ├─47565 php-fpm: master process (/etc/php-fpm.conf
           ├─47566 php-fpm: pool www
           ├─47567 php-fpm: pool www
           ├─47568 php-fpm: pool www
           ├─47569 php-fpm: pool www
           └─47570 php-fpm: pool www
--2016-04-08 13:05:08--  http://10.2.1.11/fog//management/export.php?type=sql
Connecting to 10.2.1.11:80... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2016-04-08 13:05:09 ERROR 500: Internal Server Error.

Tom Elliott

try updating.

Wayne Workman

Whatever the issue was, it’s fixed in r7118.

Tom Elliott

@Wayne-Workman it was a switch from self to static. In the case of the base class everything needs to be self.

build 7102 'Backing up database ... failed'

184

12.0k

17.3k

155.2k