FOG's web interface doesn't work.

[Wayne Workman]

Server

• FOG Version: 1.3.0 RC-21
• OS: CentOS 7

Description

The apache test page works fine when I browse to the main server’s IP address.
MySQL is running and working, apache is running and working.
SELinux is permissive
firewalld is off.
No partitions are full.

This error is spamming the apache error logs:

[Tue Nov 08 15:29:36.137517 2016] [:error] [pid 4720] [client 10.51.1.53:59564] PHP Fatal error:  Uncaught exception 'Exception' with message 'Invalid MAC Address!' in /var/www/html/fog/lib/fog/wakeonlan.class.php:54\nStack trace:\n#0 /var/www/html/fog/lib/fog/fogpage.class.php(3150): WakeOnLan->send()\n#1 /var/www/html/fog/lib/fog/fogpagemanager.class.php(253): FOGPage->wakeEmUp()\n#2 /var/www/html/fog/management/index.php(49): FOGPageManager->render()\n#3 {main}\n  thrown in /var/www/html/fog/lib/fog/wakeonlan.class.php on line 54

[Wayne Workman]

After trying to roll back to RC-20, backing up the DB fails, and I looked in the installer log, the web call just times out.

that caused me to look in apache logs, and I find these:

[Tue Nov 08 15:43:07.115278 2016] [:error] [pid 13245] [client 10.34.8.14:50703] PHP Warning:  PDOStatement::execute(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 533
[Tue Nov 08 15:43:07.115319 2016] [:error] [pid 13245] [client 10.34.8.14:50703] PHP Warning:  PDOStatement::execute(): Error reading result set's header in /var/www/html/fog/lib/db/pdodb.class.php on line 533
[Tue Nov 08 15:43:07.117335 2016] [:error] [pid 13605] [client 10.26.3.32:51435] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Nov 08 15:43:07.148999 2016] [:error] [pid 13609] [client 10.34.8.14:50704] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Nov 08 15:43:07.195836 2016] [:error] [pid 13243] [client 10.26.3.32:51436] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Nov 08 15:43:07.231981 2016] [:error] [pid 11670] [client 10.34.8.14:50705] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Nov 08 15:43:07.894371 2016] [:error] [pid 13607] [client 10.13.8.150:50615] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Nov 08 15:43:07.924605 2016] [:error] [pid 12302] [client 10.13.8.150:50616] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Nov 08 15:43:08.017120 2016] [:error] [pid 11680] [client 10.13.8.150:50617] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Nov 08 15:43:08.085943 2016] [:error] [pid 13246] [client 10.13.8.150:50618] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Nov 08 15:43:14.913975 2016] [mpm_prefork:notice] [pid 11551] AH00170: caught SIGWINCH, shutting down gracefully
[Tue Nov 08 15:43:14.916920 2016] [:error] [pid 11831] [client 10.2.3.28:51206] PHP Warning:  PDOStatement::execute(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 533
[Tue Nov 08 15:43:14.916957 2016] [:error] [pid 11831] [client 10.2.3.28:51206] PHP Warning:  PDOStatement::execute(): Error reading result set's header in /var/www/html/fog/lib/db/pdodb.class.php on line 533
[Tue Nov 08 15:43:14.948713 2016] [:error] [pid 11671] [client 10.2.3.28:51469] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Nov 08 15:45:29.899982 2016] [core:notice] [pid 17134] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Tue Nov 08 15:45:29.901262 2016] [suexec:notice] [pid 17134] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Nov 08 15:45:29.940149 2016] [auth_digest:notice] [pid 17134] AH01757: generating secret for digest authentication ...
[Tue Nov 08 15:45:29.940981 2016] [lbmethod_heartbeat:notice] [pid 17134] AH02282: No slotmem from mod_heartmonitor
[Tue Nov 08 15:45:29.966820 2016] [mpm_prefork:notice] [pid 17134] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.6.27 configured -- resuming normal operations
[Tue Nov 08 15:45:29.966873 2016] [core:notice] [pid 17134] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Tue Nov 08 15:45:29.999668 2016] [:error] [pid 17143] [client 10.32.8.37:51416] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.025903 2016] [:error] [pid 17144] [client 10.11.8.115:50358] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.162753 2016] [:error] [pid 17145] [client 10.25.3.39:51919] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.176315 2016] [:error] [pid 17141] [client 10.27.3.53:50561] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.183935 2016] [:error] [pid 17143] [client 10.13.8.91:53195] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.195515 2016] [:error] [pid 17144] [client 10.11.8.141:55536] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.221170 2016] [:error] [pid 17145] [client 10.13.8.202:56853] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.291688 2016] [:error] [pid 17141] [client 10.31.3.36:50684] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.320926 2016] [:error] [pid 17145] [client 10.13.32.35:50845] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.348060 2016] [:error] [pid 17141] [client 10.33.8.37:50690] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431
[Tue Nov 08 15:45:30.742359 2016] [:error] [pid 17144] [client 10.57.5.1:51049] PHP Fatal error:  Call to a member function getDB() on null in /var/www/html/fog/commons/init.php on line 431

[JJ Fullmer]

Did you try re-running the RC21 installer?
I’m having an issue with the RC21 gui but it does mostly work and shows up and such.
The error message appears to have something to do with wakeonlan and an invalid mac address. Maybe there’s a wake-on-lan related task that’s stuck on that 10.51.1.53 client?
Maybe try manually clearing the active tasks in the database?

[Wayne Workman]

@JJ-Fullmer I truncated the tasks table already. Forgot to mention that. And I’ve tried re-installing RC-21 also before trying to go back to RC-20.

[JJ Fullmer]

@Wayne-Workman Well then what is that host 10.51.1.53? is it your server or a random client?

[Wayne Workman]

@JJ-Fullmer it’s the main fog server.

[Wayne Workman]

This post is deleted!

[Wayne Workman]

So after extensive troubleshooting I finally figured what was causing this.

A single storage node being totally offline and unreachable.

Via CLI through MySQL, I disabled that one node, and all was well.

I’ve told Tom this, and he’s working on what caused the web interface to simply not work due to a down storage node.

[Tom Elliott]

I believe your gui response time was horrible. I don’t know why, if that’s the only node that’s “enabled” persay.

The graph enabled plays little bounds in regards to the client count (yes I know).

If the graph is disabled, it doesn’t mean the slots aren’t available. So I have a more automated means of checking these things. When it comes to dashboard getting/setting specifics, we only care about the is graph enabled, (in regards to bandwidth and graph size).

Running through many different components, if you can try accessing a direct link as opposed to starting off in the “home” page, do things actually seem to work?.

I’ve, in regards to this, figured out a more appropriate way handle the successive connections. No a single item being down should not have any impact on page display and I’m completely unable to validate. I now have a much cleaner way to handle the url processing though.

This is what fixed the GUI unable to deploy issue for @JJ-Fullmer . I am only guessing here though.

The page loads tons faster now and the javascript will help prevent tons of access requests to the other nodes as well. (We still test the active count which still requires the checks though) so some connections will have to be made. If that isn’t “suitable” then maybe we should remove active count auto-updating? It is currently set to update at the same interval as bandwidth page.

[Wayne Workman]

@Tom-Elliott

The graph enabled plays little bounds in regards to the client count (yes I know).

The node that was totally in the real world offline - it’s graph was disabled but the node was enabled.

All I did was change the node from enabled to disabled. Immediately the web interface started functioning.

Later, I re-enable the node (not the nodes graph) and immediately tge web interface totally quit working again.

[Tom Elliott]

@Wayne-Workman im aware of this, but I have still yet to see this myself. The problem, as I stated earlier, is it still is checking all nodes and for what ever reason is hanging on the down node. Was it white screen or seeming to try loading? (Circling about) which was why I asked about going directly to a site such as node =host.

What are your URL timeouts set to?

[Wayne Workman]

@Tom-Elliott my url timeouts are whatever is the default. I messed with that earlier without any effect so I put it back as it was.

The page attempted to load for a while, quite a good while. Perhaps 20 or so seconds. Then it failed.

[Tom Elliott]

@Wayne-Workman maybe 30 seconds, without fail? Default execution time for php is typically 30 seconds. Then it will fatally die out. I’ll try working out the kinks a bit later, such as having JavaScript handing if the page is available or not.

[Tom Elliott]

RC-22 officially released.

Please update, though for your unable to use issue I don’t know fully a nice answer.

I still haven’t been able to replicate.

[Wayne Workman]

I’ve updated to RC-23. I can’t say if the issue is solved or not because all of our nodes are online right now.

[Tom Elliott]

If you get a chance, maybe try stopping the httpd/apache2 services on a couple of nodes to see if all just get’s hung back up?

[Tom Elliott]

Any word on this?

[Wayne Workman]

I think the issue is solved. At home, I just shutdown one of the nodes I have without adjusting any configuration and the web interface stayed operational.