503 Service Unavailable Error

[george1421]

@troye-johnson This is an interesting one. Take a peek at the apache error log in /var/log/httpd.

Go to the management console to get the 500 error and then just tail the error log file and see what apache isn’t happy with.

[Greg Plamondon]

I am getting this as well… its hit and miss. If I refresh the page the web interface comes back.

[george1421]

@greg-plamondon OK let me update my dev box (its still on 1.5.0) to see if I can find anything.

[Troye Johnson]

[Wed Apr 11 14:06:00.009232 2018] [proxy_fcgi:error] [pid 2887] (70007)The timeout specified has expired: [client x.x.x.x:56333] AH01075: Error dispatching request to :

Is the error i am receiving.

[Greg Plamondon]

I found this in the apache logs when it happens.

[Wed Apr 11 13:52:45.028047 2018] [proxy_fcgi:error] [pid 2841] (70007)The timeout specified has expired: [client 10.10.150.73:62324] AH01075: Error dispatching request to :, referer: http://10fogserver/fog/management/index.php?node=host&sub=deploy&id=160&type=1

[george1421]

@greg-plamondon said in 503 Service Unavailable Error:

proxy_fcgi

That kind of tells me that php-fpm was enabled and its having an issue. What OS do you have Greg? My dev box is centos so I should be able to duplicate Troye’s issue.

I wasn’t aware that the devs enabled php-fpm (which is generally a good thing to have running).

[Greg Plamondon]

@george1421 I am running Centos 7.4 x64

[george1421]

(still waiting for my dev server to update)
I did do some testing for php-fpm a while ago: https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast

The idea of php-fpm is a dedicated php engine that executes much faster than the built in php engine in apache.

[george1421]

@george1421 ok not related to php-fpm.

How many client computers do you guys have on your campus?

[Troye Johnson]

I have about 130-150 clients.

[Wayne Workman]

@greg-plamondon said in 503 Service Unavailable Error:

I found this in the apache logs when it happens.

[Wed Apr 11 13:52:45.028047 2018] [proxy_fcgi:error] [pid 2841] (70007)The timeout specified has expired: [client 10.10.150.73:62324] AH01075: Error dispatching request to :, referer: http://10fogserver/fog/management/index.php?node=host&sub=deploy&id=160&type=1

Did you purposely put 10fogserver in there?

[george1421]

@george1421 OK I’m back hunting php-fpm timeout.

For those with the error can you review this log file to see if it gives us any clue to why /var/log/php-fpm/www-error.log

[Greg Plamondon]

@george1421
I followed your FPM/FastCGI howto you linked and the problem seems to be gone and considerably faster now.
Thanks!

[Troye Johnson]

I will try your guide.

[Greg Plamondon]

@wayne-workman Yes, as we have more than one.

[george1421]

@george1421 said in 503 Service Unavailable Error:
For those with the error can you review this log file to see if it gives us any clue to why /var/log/php-fpm/www-error.log

Its great we have php-fpm running, it does make quite a difference, but we need to understand why. I can’t seem to make it fail on my side, but I only have a handful of systems on my dev box.

[Greg Plamondon]

@george1421 said in 503 Service Unavailable Error:

@george1421 said in 503 Service Unavailable Error:
For those with the error can you review this log file to see if it gives us any clue to why /var/log/php-fpm/www-error.log

Its great we have php-fpm running, it does make quite a difference, but we need to understand why. I can’t seem to make it fail on my side, but I only have a handful of systems on my dev box.

this is my log:

[11-Apr-2018 14:53:04 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 35258 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 14:53:14 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 35258 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 14:53:27 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 35258 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 15:30:55 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 15:33:22 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:00:49 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:08:10 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:13:25 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:13:46 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:14:03 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:14:28 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:14:31 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:16:21 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:17:13 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:17:30 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:17:51 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:21:05 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:21:28 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:22:15 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:24:21 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:24:57 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:25:19 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:25:45 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:26:28 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:44:16 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:44:39 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:44:54 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:45:19 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:46:00 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:46:58 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:47:54 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:48:13 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:48:23 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:48:52 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:49:06 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:49:30 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260

[george1421]

@greg-plamondon Ok it looks like an upper memory limit got us. OK lets see what we need to do to fix. @Troye-Johnson since Greg already tainted his system hold off for a minute updating yours so we can see if we can fix this memory issue.

[Tom Elliott]

@george1421 change the memory limit it fogsettings

[Troye Johnson]

here is some information I got from the error.log

[10-Apr-2018 12:36:06] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 2 idle, and 11 total children
[10-Apr-2018 12:36:07] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 3 idle, and 14 total children
[10-Apr-2018 12:36:08] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 4 idle, and 16 total children
[10-Apr-2018 12:36:09] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 4 idle, and 17 total children
[11-Apr-2018 13:21:32] NOTICE: Terminating …
[11-Apr-2018 13:21:32] NOTICE: exiting, bye-bye!
[11-Apr-2018 13:23:53] NOTICE: fpm is running, pid 3560
[11-Apr-2018 13:23:53] NOTICE: ready to handle connections
[11-Apr-2018 13:23:53] NOTICE: systemd monitor interval set to 10000ms
[11-Apr-2018 13:32:55] NOTICE: Terminating …
[11-Apr-2018 13:32:55] NOTICE: exiting, bye-bye!
[11-Apr-2018 13:33:36] NOTICE: fpm is running, pid 969
[11-Apr-2018 13:33:36] NOTICE: ready to handle connections
[11-Apr-2018 13:33:36] NOTICE: systemd monitor interval set to 10000ms