High CPU usage from multiple httpd processes version 5800



  • Hi all

    running version 5800 - noticed today loads of httpd processes running on my main FOG server and CPU was overloading. Similar to this post I noticed it due to alarms in VMWare.

    I’ve updated to latest git version (6679) but still showing the same high CPU/multiple httpd processes.

    I have a second FOG server which was on same version which does not exhibit this issue.

    Is this linked in some way to the new FOG client calling the server? I have 4 users in our Service Desk who access the web GUI and even with no-one connected to the GUI I’m still seeing all these httpd connections.

    httpd access and error logs not telling me much, anywhere else I can look to try and diagnose this issue?

    cheers, Kiweegie.


  • Senior Developer

    @Sebastian-Roth The real question is the original issue that caused this post to be opened in the first place fixed?

    @Kiweegie, are you solving this because the password/snapin issue is fixed or your server is no longer sufferring from the High CPU Usage from multiple httpd processes?


  • Developer

    @Wayne-Workman Should have reloaded the page… ;-) Good to know that you seam to have verified that this is fixed. Probably it was still wrong in .fogsettings for @Kiweegie?!



  • @Wayne-Workman HI Wayne thanks for the heads up. fog password in this case is purely alphanumeric though so don’t think that was it. Resetting the fog password has resolved my issue however so happy to have this one marked as solved.

    regards Kiweegie.


  • Developer

    @Kiweegie I vaguely remember @Wayne-Workman having seen a fog user password change after running the installer script at some point. Maybe this is related? https://forums.fogproject.org/topic/6649/storage-node-installation-resets-local-fog-user-s-password-to-something-wrong

    Cannot actually find the commit that was made to fix this but from what Tom says I thought this was fixed somewhere around version 63xx. Maybe not? Please keep an eye on this and let us know!


  • Moderator

    @Kiweegie The installer a week to two weeks ago set it wrong if there were certain special characters present in the password. It’s since been fixed though. You should verify that what is inside of .fogsettings is correct, and what shows up for the user/pass in the storage management area is correct, and for the kernel update stuff too. All of this is outlined in the “Troubleshoot FTP” wiki article.



  • @Tom-Elliott Hi Tom unfortunately I was going off the information passed to me by my service desk… which turns out to not be as accurate as I thought :(

    I’ve tried deploying snapin and while there is no error thrown up in the GUI, c:\fog.log on target machine shows the same error as seen when trying to deploy image.

     14/03/2016 13:35 Middleware::Communication Unknown Response: Type: 2, File: /var/www/html/fog/lib/fog/fogftp.class.php, Line: 104, Message: ftp_login(): Login incorrect.
    

    This would suggest that the fog or ftp user password has been changed but it has not been altered at least not by me. Entry in following 2 locations matched what I had in keepass

    Storage Management > All Storage nodes > DefaultMember > Management Password
    FOG Configuration > FOG Settings > TFTP Server > FOG_TFTP_FTP_Password

    However, resetting the fog user password at commandline fixed the issue…

    Don’t understand who or what amended that password as only I have access to the box and it sure wasn’t me!

    cheers . Kiweegie.



  • @george1421 said:

    FOG Configuration->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME

    cheers George!


  • Moderator

    @Kiweegie FOG Configuration->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME


  • Senior Developer

    @Kiweegie FOG Configuration Page->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME,FOG_GRACE_TIMEOUT



  • @Wayne-Workman @george1421 Hi Wayne/George

    Where do I find the client checkin time setting? I’ve checked fog settings and nothing shows up there. Could you point me in the right direction please?

    cheers Kiweegie


  • Senior Developer

    @Kiweegie I’m not able to replicate this. If snapins are working off of the same nodes that are not sending the image, then is something potentially wrong with how the images are mounted on that system? I know it’s stating an FTP error, so it’s a wee bit odd.



  • @Tom-Elliott Morning Tom not sure if related to changes you made for me late last week or a new issue but since the teamviewer session we had, deployment of images fails with error below:

    failed to create deployment tasking....

    The offending line in path /var/www/html/fog/lib/fog/fogftpclass.php is on line 104

          if (@ftp_login(self::$link,$username,$password) === false) self::ftperror();
    

    no credentials have changed at all. Snapins deploy fine, only images are affected and we have tried with multiple images.

    regards, Kiweegie.


  • Moderator

    FWIW, I have mine set to 900 seconds. I don’t deploy snapins post deployment or issue unattended deployments. There is no real value (in my environment) to have a client check-ins faster than every 15 minutes.


  • Moderator

    @Kiweegie Turn down your client checkin time. At work, I’ve got mine set to 300 seconds. IMHO, 60 seconds is absurd.


  • Senior Developer

    @Kiweegie There is a couple things the reach out to the internet, one is the FOG Services (Scheduler, MulticastManager, etc…) that reach out to the internet to get the system’s external address in the case that’s the way your node is setup to be reached, so I doubt those are the “problem” things.



  • @Tom-Elliott Nope, not without vpn to our corporate network


  • Senior Developer

    @Kiweegie is your fog server reachable from the Internet?



  • Possible red-herring but worthwhile raising the question I think.

    watching connections via

    ss -tp
    

    throws up amongst a host of other connections

    CLOSE-WAIT 1      0           Fog-server-IP:45260        54.93.153.156:http
    CLOSE-WAIT 1      0           Fog-server-IP:45856        54.93.153.156:http
    
    C:\Windows\System32>nslookup 54.93.153.156
    Server:  my_AD_Domain controller
    Address:   my_AD_Domain controller_IP
    
    Name:    ec2-54-93-153-156.eu-central-1.compute.amazonaws.com
    Address:  54.93.153.156
    

    for what reason would FOG be looking to connect to an Amazon IP??? I’m seeing a similar connection attempt to 52.28.249.93 which is another Amazon IP.

    There are a load of ESTABLISHED connections showing like below which I believe to be the actual httpd processes though.

    Proto Recv-Q Send-Q Local Address                Foreign Address             State
    tcp        0      0 ::ffff:Fof_Server_IP:80     ::ffff:Client-IP:64981 ESTABLISHED
    

    If you need any other logs etc to help get to the bottom of this I’m more than happy to oblige.

    cheers Kiweegie.



  • @Tom-Elliott Sorry got sidetracked by lunch :)

    Still several (40+) httpd processes running. FOGScheduler service is off.

    Checking /var/log/httpd/access_log shows the following connections from multiple clients

    10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/snapins.checkin.php?mac=D4:3D:7E:25:49:7F%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 4 "-" "-"
    10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/Printers.php?mac=D4:3D:7E:38:FE:60%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 12 "-" "-"
    10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/greenfog.php?mac=74:27:EA:AD:AC:0A%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 73 "-" "-"
    10.***.***.***- - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/autologout.php?mac=A4:1F:72:86:3A:A4%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
    10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/autologout.php?mac=74:27:EA:CF:9A:FA%7C%7C00:00:00:00:00:00:00:E0%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
    10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/servicemodule-active.php?moduleid=taskreboot&mac=A4:1F:72:86:5C:CB%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 5 "-" "-"
    10.***.***.*** - - [11/Mar/2016:14:26:51 +0000] "GET /fog/service/autologout.php?mac=A4:1F:72:86:5C:CB%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
    10.***.***.*** - - [11/Mar/2016:14:26:53 +0000] "GET /fog/status/bandwidth.php?dev=eth0 HTTP/1.1" 200 47 "-" "-"
    10.***.***.*** - - [11/Mar/2016:14:26:51 +0000] "GET /fog/service/printerlisting.php?newService=1 HTTP/1.1" 200 5 "-" "-"
    

    And in /var/log/httpd/error_log seeing this

    [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::real_connect(): (08004/1040): Too many connections in /var/www/html/fog/lib/db/mysql.class.php on line 30
    [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::set_charset(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 31
    [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
    [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
    [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 52
    [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::real_connect(): (08004/1040): Too many connections in /var/www/html/fog/lib/db/mysql.class.php on line 30
    [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::set_charset(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 31
    [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
    [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
    [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 52
    df: `/home/fog/images': Permission denied
    df: no file systems processed
    [Fri Mar 11 13:22:15 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
    [Fri Mar 11 13:22:17 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
    [Fri Mar 11 13:22:19 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
    [Fri Mar 11 13:22:21 2016] [error] child process 2991 still did not exit, sending a SIGKILL
    [Fri Mar 11 13:22:22 2016] [notice] caught SIGTERM, shutting down
    [Fri Mar 11 13:23:30 2016] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
    [Fri Mar 11 13:23:30 2016] [notice] Digest: generating secret for digest authentication ...
    [Fri Mar 11 13:23:30 2016] [notice] Digest: done
    [Fri Mar 11 13:23:31 2016] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.6.19 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
    df: `/home/fog/images': Permission denied
    df: no file systems processed
    

    From another post I’ve looked at max connections in mysql (this is a CentOS 6.7 server)

    su mysql
    ulimit -n
    1024
    
    ulimit -a
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 7414
    max locked memory       (kbytes, -l) 64
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 1024
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 1024
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited
    

    Does this look ok? Should I look to increase the max connections above 1024?

    regards Kiweegie.


Log in to reply
 

384
Online

38974
Users

10712
Topics

101674
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.