High CPU, PHP Errors after update to trunk (github) 7234
-
I probably should have posted this in the bug reports section. Please move it if we need to do that.
Updated to 7238, and then 7240 today. Nagios started complaining about sustained high CPU load, and there are several pages that aren’t working.
Here is some info from top:
29419 www-data 20 0 326492 17936 8156 S 28.8 0.1 0:11.29 apache2 29399 www-data 20 0 326844 17476 7728 R 28.4 0.1 0:16.13 apache2 29337 www-data 20 0 326740 17416 7776 R 27.3 0.1 0:34.32 apache2 29444 www-data 20 0 326568 17192 7492 R 27.3 0.1 0:04.13 apache2 24973 mysql 20 0 3450612 84784 7220 S 27.0 0.5 6:11.74 mysqld 29050 www-data 20 0 476664 19868 9528 R 27.0 0.1 1:51.93 apache2 29254 www-data 20 0 326476 17548 7936 S 27.0 0.1 0:58.32 apache2 29306 www-data 20 0 326096 18144 8808 R 27.0 0.1 0:48.90 apache2 29415 www-data 20 0 326872 17652 7648 S 26.6 0.1 0:11.11 apache2 29416 www-data 20 0 326728 16992 7356 R 26.6 0.1 0:10.60 apache2 29446 www-data 20 0 326588 17324 7600 S 26.6 0.1 0:04.13 apache2 29456 www-data 20 0 324252 14568 5656 R 26.3 0.1 0:01.33 apache2 29445 www-data 20 0 326824 17316 7588 R 25.5 0.1 0:03.77 apache2 29400 www-data 20 0 326316 17248 7824 S 25.2 0.1 0:16.12 apache2 29395 www-data 20 0 326304 17152 8028 S 24.8 0.1 0:17.92 apache2 29401 www-data 20 0 326044 17060 7896 S 24.8 0.1 0:16.08 apache2 29407 www-data 20 0 326284 16768 7672 S 24.1 0.1 0:12.86 apache2 29461 www-data 20 0 325948 16064 7136 S 20.9 0.1 0:00.58 apache2And here are some entries from /var/log/apache2/error.log:
[Wed Apr 20 14:44:38.384617 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host [Wed Apr 20 14:44:38.385009 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host [Wed Apr 20 14:44:38.387353 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host [Wed Apr 20 14:44:38.387844 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host [Wed Apr 20 14:44:38.388283 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host [Wed Apr 20 14:44:38.389913 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host [Wed Apr 20 14:44:38.402290 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host [Wed Apr 20 14:44:38.404902 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host [Wed Apr 20 14:44:38.406225 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host [Wed Apr 20 14:44:38.406687 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=hostAlso seeing this in the log file:
Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /var/www/fog/lib/db/mysql.class.php on line 67, referer: http://fog/fog/management/index.php?node=reportIn case the version of PHP is relevant, php -v returns:
PHP 5.6.20-1+deb.sury.org~trusty+1 (cli) Copyright (c) 1997-2016 The PHP Group Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies -
We’ve seen this before. I think your MySQL connections hit the maximum, and you need to raise it. Look at this:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_MySQL -
@apathetic_admin How many clients do you have??
-
@Sebastian-Roth 650ish.
-
@Wayne-Workman I made that change, unfortunately it does not seem to have made an impact.
I also just tried to run a report and got this gem:
[Thu Apr 21 09:30:03.003142 2016] [:error] [pid 29602] [client 10.128.7.34:52444] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /var/www/fog/lib/db/mysql.class.php on line 67, referer: http://fog/fog/management/index.php?node=report [Thu Apr 21 09:30:03.073369 2016] [:error] [pid 29602] [client 10.128.69.110:54425] PHP Fatal error: Cannot redeclare class Initiator in /var/www/fog/commons/init.php on line 2 [Thu Apr 21 09:30:04.125460 2016] [core:notice] [pid 25642] AH00051: child pid 29602 exit signal Segmentation fault (11), possible coredump in /etc/apache2 -
Then let’s try increasing PHP’s memory limit: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_Web_Interface
-
Spent some time with Tom and I think it’s better now. We rolled back and fourth through various revisions, and then went to php 7, and the load averages are a bit lower than they were before. The report issue was fixed by upping the memory limit for FOG in the FOG settings.
-
@apathetic_admin May I ask if you still know which version you used before the update? Maybe see the install logs?? You didn’t have any load problems before the upgrade?
-
@Sebastian-Roth he was on version 6152.