503 Service Unavailable Error

[Wayne Workman]

@greg-plamondon said in 503 Service Unavailable Error:

I found this in the apache logs when it happens.

[Wed Apr 11 13:52:45.028047 2018] [proxy_fcgi:error] [pid 2841] (70007)The timeout specified has expired: [client 10.10.150.73:62324] AH01075: Error dispatching request to :, referer: http://10fogserver/fog/management/index.php?node=host&sub=deploy&id=160&type=1

Did you purposely put 10fogserver in there?

[george1421]

@george1421 OK I’m back hunting php-fpm timeout.

For those with the error can you review this log file to see if it gives us any clue to why /var/log/php-fpm/www-error.log

[Greg Plamondon]

@george1421
I followed your FPM/FastCGI howto you linked and the problem seems to be gone and considerably faster now.
Thanks!

[Troye Johnson]

I will try your guide.

[Greg Plamondon]

@wayne-workman Yes, as we have more than one.

[george1421]

@george1421 said in 503 Service Unavailable Error:
For those with the error can you review this log file to see if it gives us any clue to why /var/log/php-fpm/www-error.log

Its great we have php-fpm running, it does make quite a difference, but we need to understand why. I can’t seem to make it fail on my side, but I only have a handful of systems on my dev box.

[Greg Plamondon]

@george1421 said in 503 Service Unavailable Error:

@george1421 said in 503 Service Unavailable Error:
For those with the error can you review this log file to see if it gives us any clue to why /var/log/php-fpm/www-error.log

Its great we have php-fpm running, it does make quite a difference, but we need to understand why. I can’t seem to make it fail on my side, but I only have a handful of systems on my dev box.

this is my log:

[11-Apr-2018 14:53:04 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 35258 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 14:53:14 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 35258 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 14:53:27 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 35258 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 15:30:55 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 15:33:22 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:00:49 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:08:10 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:13:25 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:13:46 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:14:03 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:14:28 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:14:31 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:16:21 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:17:13 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:17:30 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:17:51 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:21:05 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:21:28 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:22:15 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:24:21 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:24:57 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:25:19 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:25:45 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 16:26:28 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:44:16 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:44:39 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:44:54 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:45:19 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:46:00 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:46:58 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:47:54 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:48:13 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:48:23 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:48:52 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:49:06 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260
[11-Apr-2018 17:49:30 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 260

[george1421]

@greg-plamondon Ok it looks like an upper memory limit got us. OK lets see what we need to do to fix. @Troye-Johnson since Greg already tainted his system hold off for a minute updating yours so we can see if we can fix this memory issue.

[Tom Elliott]

@george1421 change the memory limit it fogsettings

[Troye Johnson]

here is some information I got from the error.log

[10-Apr-2018 12:36:06] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 2 idle, and 11 total children
[10-Apr-2018 12:36:07] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 3 idle, and 14 total children
[10-Apr-2018 12:36:08] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 4 idle, and 16 total children
[10-Apr-2018 12:36:09] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 4 idle, and 17 total children
[11-Apr-2018 13:21:32] NOTICE: Terminating …
[11-Apr-2018 13:21:32] NOTICE: exiting, bye-bye!
[11-Apr-2018 13:23:53] NOTICE: fpm is running, pid 3560
[11-Apr-2018 13:23:53] NOTICE: ready to handle connections
[11-Apr-2018 13:23:53] NOTICE: systemd monitor interval set to 10000ms
[11-Apr-2018 13:32:55] NOTICE: Terminating …
[11-Apr-2018 13:32:55] NOTICE: exiting, bye-bye!
[11-Apr-2018 13:33:36] NOTICE: fpm is running, pid 969
[11-Apr-2018 13:33:36] NOTICE: ready to handle connections
[11-Apr-2018 13:33:36] NOTICE: systemd monitor interval set to 10000ms

[Troye Johnson]

@troye-johnson

[10-Apr-2018 13:41:10 UTC] PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /var/www/html/fog/lib/db/mysqldump.class.php on line 383
[10-Apr-2018 13:41:12 UTC] PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /var/www/html/fog/lib/db/mysqldump.class.php on line 400
[10-Apr-2018 14:07:13 UTC] PHP Warning:  Error while sending QUERY packet. PID=3131 in /var/www/html/fog/lib/db/pdodb.class.php on line 591
[10-Apr-2018 14:07:13 UTC] PHP Warning:  array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 1095
[10-Apr-2018 14:07:14 UTC] PHP Warning:  Error while sending QUERY packet. PID=1228 in /var/www/html/fog/lib/db/pdodb.class.php on line 591
[10-Apr-2018 14:08:08 UTC] PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /var/www/html/fog/lib/db/mysqldump.class.php on line 383
[10-Apr-2018 14:08:09 UTC] PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /var/www/html/fog/lib/db/mysqldump.class.php on line 400
[11-Apr-2018 07:13:07 UTC] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/html/fog/lib/fog/fogpagemanager.class.php on line 386
[11-Apr-2018 07:13:07 UTC] PHP Fatal error:  Uncaught exception 'Exception' with message '#!im' in /var/www/html/fog/lib/fog/fogbase.class.php:584
Stack trace:
#0 /var/www/html/fog/lib/client/registerclient.class.php(47): FOGBase::getHostItem(true, false, false, true)
#1 /var/www/html/fog/lib/fog/fogpage.class.php(3008): RegisterClient->json()
#2 /var/www/html/fog/lib/fog/fogpage.class.php(249): FOGPage->requestClientInfo()
#3 /var/www/html/fog/lib/pages/processlogin.class.php(57): FOGPage->__construct('')
#4 /var/www/html/fog/lib/fog/fogpagemanager.class.php(381): ProcessLogin->__construct()
#5 [internal function]: FOGPageManager->{closure}('/var/www/html/f...')
#6 /var/www/html/fog/lib/fog/fogpagemanager.class.php(386): array_map(Object(Closure), Array)
#7 /var/www/html/fog/lib/fog/fogpagemanager.class.php(79): FOGPageManager->loadPageClasses()
#8 [internal function]: FOGPageManager->__construct()
#9 /var/www/html/fog/lib/fog/fogbase.class.php(481): ReflectionClass->newInstanceArgs(Array)
#10 /var/www/html/fog/management/index.php(23): FOGBase::getCla in /var/www/html/fog/lib/fog/fogbase.class.php on line 584
[11-Apr-2018 17:24:03 UTC] PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /var/www/html/fog/lib/db/mysqldump.class.php on line 383
[11-Apr-2018 17:24:06 UTC] PHP Warning:  date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /var/www/html/fog/lib/db/mysqldump.class.php on line 400
[11-Apr-2018 19:00:56 UTC] PHP Warning:  phpinfo(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /var/www/html/info.php on line 5

[george1421]

@tom-elliott said in 503 Service Unavailable Error:

@george1421 change the memory limit it fogsettings

I’m just trying to understand where its getting the 134MB value from. PHP-FPM should read the php.ini for the memory limit (which is 128MB) unless it is overridden in its own config file (which is commented out at the moment).

[Troye Johnson]

I still can not get the web page to work.

how would I go about changing the memory limit in the fog settings if I cannot get to the settings page?

[george1421]

@troye-johnson I’m not so concerned about the timezone messages, but the rest of them look pretty interesting. Your issue doesn’t seem to be a memory issue as with Greg’s. Your’s looks a bit more troubling.

[george1421]

@troye-johnson its in /etc/php.ini search for memory

The value looks like this:

memory_limit = 128M

Change it to 256M save it then reboot the fog server. I really don’t think your issue is memory since the log file doesn’t say anything about memory exhaustion.

[george1421]

@greg-plamondon Greg, if you still have memory exhaustion issues with my config I want you to change the memory allocation in the www.conf file. We need you guys to help us debug this issue.

I’m not getting a clear answer (my google-fu is weak today) on if php-fpm looks at the php.ini config file for its memory settings, or uses its own.

[Tom Elliott]

The fog settings overrides the php ini setting. So you can set whatever you want in the php.ini, but the fog settings overrides it. This is intentional. This way you can limit when memory usage should be expanded to only usage on fog.

[Troye Johnson]

changing the limit did not help still getting the 503 error.

[Tom Elliott]

https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast/3

Please try following this in regards to your 503 error.

The memory limit wouldn’t play an impact on the 503, it’d most likely do a 500 as it would crash.

[Greg Plamondon]

ok, I thought it was fixed for me but I still get the 503 when doploying an image.
If i refresh and look in tasks the task was created.

here is my httpd error.log

[Wed Apr 11 15:44:33.208616 2018] [proxy_fcgi:error] [pid 31003:tid 140654097147648] (70007)The timeout specified has expired: [client 192.168.10.170:50376] AH01075: Error dispatching request to :, referer: http://10fogserver.mtstrans.com/fog/management/index.php?node=host&sub=deploy&id=698&type=1

I dont want to Hijack @Troye-Johnson thread… Should a create a new one?