High CPU, PHP Errors after update to trunk (github) 7234



  • I probably should have posted this in the bug reports section. Please move it if we need to do that.

    Updated to 7238, and then 7240 today. Nagios started complaining about sustained high CPU load, and there are several pages that aren’t working.

    Here is some info from top:

    29419 www-data  20   0  326492  17936   8156 S  28.8  0.1   0:11.29 apache2                                                                                                                                                                                    
    29399 www-data  20   0  326844  17476   7728 R  28.4  0.1   0:16.13 apache2                                                                                                                                                                                    
    29337 www-data  20   0  326740  17416   7776 R  27.3  0.1   0:34.32 apache2                                                                                                                                                                                    
    29444 www-data  20   0  326568  17192   7492 R  27.3  0.1   0:04.13 apache2                                                                                                                                                                                    
    24973 mysql     20   0 3450612  84784   7220 S  27.0  0.5   6:11.74 mysqld                                                                                                                                                                                     
    29050 www-data  20   0  476664  19868   9528 R  27.0  0.1   1:51.93 apache2                                                                                                                                                                                    
    29254 www-data  20   0  326476  17548   7936 S  27.0  0.1   0:58.32 apache2                                                                                                                                                                                    
    29306 www-data  20   0  326096  18144   8808 R  27.0  0.1   0:48.90 apache2                                                                                                                                                                                    
    29415 www-data  20   0  326872  17652   7648 S  26.6  0.1   0:11.11 apache2                                                                                                                                                                                    
    29416 www-data  20   0  326728  16992   7356 R  26.6  0.1   0:10.60 apache2                                                                                                                                                                                    
    29446 www-data  20   0  326588  17324   7600 S  26.6  0.1   0:04.13 apache2                                                                                                                                                                                    
    29456 www-data  20   0  324252  14568   5656 R  26.3  0.1   0:01.33 apache2                                                                                                                                                                                    
    29445 www-data  20   0  326824  17316   7588 R  25.5  0.1   0:03.77 apache2                                                                                                                                                                                    
    29400 www-data  20   0  326316  17248   7824 S  25.2  0.1   0:16.12 apache2                                                                                                                                                                                    
    29395 www-data  20   0  326304  17152   8028 S  24.8  0.1   0:17.92 apache2                                                                                                                                                                                    
    29401 www-data  20   0  326044  17060   7896 S  24.8  0.1   0:16.08 apache2                                                                                                                                                                                    
    29407 www-data  20   0  326284  16768   7672 S  24.1  0.1   0:12.86 apache2                                                                                                                                                                                    
    29461 www-data  20   0  325948  16064   7136 S  20.9  0.1   0:00.58 apache2                                                                                                                                                                                    
    

    And here are some entries from /var/log/apache2/error.log:

    [Wed Apr 20 14:44:38.384617 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    [Wed Apr 20 14:44:38.385009 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    [Wed Apr 20 14:44:38.387353 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    [Wed Apr 20 14:44:38.387844 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    [Wed Apr 20 14:44:38.388283 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    [Wed Apr 20 14:44:38.389913 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    [Wed Apr 20 14:44:38.402290 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    [Wed Apr 20 14:44:38.404902 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    [Wed Apr 20 14:44:38.406225 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    [Wed Apr 20 14:44:38.406687 2016] [:error] [pid 29419] [client 10.128.7.15:59589] PHP Warning:  array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 196, referer: http://fog.rmgcom.local/fog/management/index.php?node=host
    

    Also seeing this in the log file:

    Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /var/www/fog/lib/db/mysql.class.php on line 67, referer: http://fog/fog/management/index.php?node=report
    

    In case the version of PHP is relevant, php -v returns:

    PHP 5.6.20-1+deb.sury.org~trusty+1 (cli) 
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
        with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    

  • Senior Developer

    @Sebastian-Roth he was on version 6152.


  • Developer

    @apathetic_admin May I ask if you still know which version you used before the update? Maybe see the install logs?? You didn’t have any load problems before the upgrade?



  • Spent some time with Tom and I think it’s better now. We rolled back and fourth through various revisions, and then went to php 7, and the load averages are a bit lower than they were before. The report issue was fixed by upping the memory limit for FOG in the FOG settings.


  • Moderator

    Then let’s try increasing PHP’s memory limit: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_Web_Interface



  • @Wayne-Workman I made that change, unfortunately it does not seem to have made an impact.

    I also just tried to run a report and got this gem:

    [Thu Apr 21 09:30:03.003142 2016] [:error] [pid 29602] [client 10.128.7.34:52444] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /var/www/fog/lib/db/mysql.class.php on line 67, referer: http://fog/fog/management/index.php?node=report
    [Thu Apr 21 09:30:03.073369 2016] [:error] [pid 29602] [client 10.128.69.110:54425] PHP Fatal error:  Cannot redeclare class Initiator in /var/www/fog/commons/init.php on line 2
    [Thu Apr 21 09:30:04.125460 2016] [core:notice] [pid 25642] AH00051: child pid 29602 exit signal Segmentation fault (11), possible coredump in /etc/apache2
    
    


  • @Sebastian-Roth 650ish.


  • Developer

    @apathetic_admin How many clients do you have??


  • Moderator

    We’ve seen this before. I think your MySQL connections hit the maximum, and you need to raise it. Look at this:
    https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_MySQL


Log in to reply
 

462
Online

38710
Users

10539
Topics

99781
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.