Storage Node Disk Usage #!db and boot.php #!db

[Wayne Workman]

Each additional node I add shows this.

We have two storage groups, both master nodes for each group are on the main server. We are using the location plugin. We have a total of 5 nodes now, this will go up to about 12 or 13 over the week.

[Wayne Workman]

So, update.

Running the installer again (without changing anything) fixes the issue.

This is actually the second time this has happened during this week.

[Tom Elliott]

#!db means it cannot connect to the database.

If you’re seeing this there’s likely a reason.

[Wayne Workman]

@Tom-Elliott If I connect from the node to the main server’s DB manually, it works, using the fogstorage credentials. I have access to the fog db and all that jazz. Also, this is the second time in a row this has happened, and the second time in a row that simply re-running the installer without any other change has fixed it.

[Tom Elliott]

@Wayne-Workman I realize that but instead of immediately rerunning the installer, which drops all connections, can you try to get a log file?

[Wayne Workman]

@Tom-Elliott Yes I’ll do that. I’ll have plenty of opportunities to do that many times this week.

[Wayne Workman]

apache errors but none seem recent enough, according to the newly displayed server time

[Tue Jun 14 14:11:44.776393 2016] [:error] [pid 22295] [client 10.51.14.15:57342] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 260
[Tue Jun 14 14:11:44.771166 2016] [:error] [pid 22295] [client 10.51.14.15:57342] PHP Warning: Error while sending QUERY packet. PID=22295 in /var/www/html/fog/lib/db/pdodb.class.php on line 169
[Tue Jun 14 14:09:44.579914 2016] [:error] [pid 18721] [client 10.51.14.15:57340] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 260
[Tue Jun 14 14:09:44.574467 2016] [:error] [pid 18721] [client 10.51.14.15:57340] PHP Warning: Error while sending QUERY packet. PID=18721 in /var/www/html/fog/lib/db/pdodb.class.php on line 169
[Tue Jun 14 14:07:44.262165 2016] [:error] [pid 18065] [client 10.51.14.15:57331] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 260
[Tue Jun 14 14:07:44.256817 2016] [:error] [pid 18065] [client 10.51.14.15:57331] PHP Warning: Error while sending QUERY packet. PID=18065 in /var/www/html/fog/lib/db/pdodb.class.php on line 169
[Tue Jun 14 14:05:44.069214 2016] [:error] [pid 18972] [client 10.51.14.15:57324] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 260
[Tue Jun 14 14:05:44.063814 2016] [:error] [pid 18972] [client 10.51.14.15:57324] PHP Warning: Error while sending QUERY packet. PID=18972 in /var/www/html/fog/lib/db/pdodb.class.php on line 169
[Tue Jun 14 14:03:43.836633 2016] [:error] [pid 18815] [client 10.51.14.15:57317] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 260
[Tue Jun 14 14:03:43.831377 2016] [:error] [pid 18815] [client 10.51.14.15:57317] PHP Warning: Error while sending QUERY packet. PID=18815 in /var/www/html/fog/lib/db/pdodb.class.php on line 169
[Tue Jun 14 14:01:43.647908 2016] [:error] [pid 22355] [client 10.51.14.15:57309] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 260
[Tue Jun 14 14:01:43.642519 2016] [:error] [pid 22355] [client 10.51.14.15:57309] PHP Warning: Error while sending QUERY packet. PID=22355 in /var/www/html/fog/lib/db/pdodb.class.php on line 169
[Tue Jun 14 13:59:43.451996 2016] [:error] [pid 18815] [client 10.51.14.15:57302] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 260

There are no error logs showing up for the new node in the web interface. I’m going to have to get them from the local server.

[Wayne Workman]

Here are apache logs from the new node, also showed that SELinux is permissive.

[root@aifog httpd]# tail error_log 
[Tue Jun 14 14:09:13.534622 2016] [core:notice] [pid 25869] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Tue Jun 14 14:09:13.536697 2016] [suexec:notice] [pid 25869] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Jun 14 14:09:13.538642 2016] [ssl:warn] [pid 25869] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using fe80::215:5dff:fe01:a00. Set the 'ServerName' directive globally to suppress this message
[Tue Jun 14 14:09:13.589443 2016] [auth_digest:notice] [pid 25869] AH01757: generating secret for digest authentication ...
[Tue Jun 14 14:09:13.591065 2016] [lbmethod_heartbeat:notice] [pid 25869] AH02282: No slotmem from mod_heartmonitor
[Tue Jun 14 14:09:13.593172 2016] [ssl:warn] [pid 25869] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Jun 14 14:09:13.677889 2016] [mpm_prefork:notice] [pid 25869] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.6.22 configured -- resuming normal operations
[Tue Jun 14 14:09:13.677928 2016] [core:notice] [pid 25869] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[root@aifog httpd]# getenforce
Permissive

[Wayne Workman]

Here’s the contents of the installer’s error_log:
0_1465932215813_fog_error_8093.log

[Wayne Workman]

@Tom-Elliott I figured it out.

The installer is writing this file wrong:
/var/www/html/fog/lib/fog/config.class.php

This line is being written at initial installation of the storage node as:
define('DATABASE_HOST','p:127.0.0.1');

When it should be:

define('DATABASE_HOST','p:x.x.x.x'); where x.x.x.x is the main server’s IP.

[Tom Elliott]

And that is fixed in latest too.