High CPU usage from multiple httpd processes version 5800

Kiweegie

Hi all

running version 5800 - noticed today loads of httpd processes running on my main FOG server and CPU was overloading. Similar to this post I noticed it due to alarms in VMWare.

I’ve updated to latest git version (6679) but still showing the same high CPU/multiple httpd processes.

I have a second FOG server which was on same version which does not exhibit this issue.

Is this linked in some way to the new FOG client calling the server? I have 4 users in our Service Desk who access the web GUI and even with no-one connected to the GUI I’m still seeing all these httpd connections.

httpd access and error logs not telling me much, anywhere else I can look to try and diagnose this issue?

cheers, Kiweegie.

Tom Elliott

Please restart the FOGScheduler service. Or stop it. It should drop the httpd processes for you. I’m still trying to figure out what’s causing IT to be the culprit here.

Kiweegie

@Tom-Elliott HI Tom - I’ve restarted the server entirely twice which didn’t help. I’ve stopped the FOGScheduler service now and watching top to see if the httpd processes disappear but so far not happening.

Will post back in 10 with further report of above.

cheers Kiweegie.

Kiweegie

@Tom-Elliott Sorry got sidetracked by lunch

Still several (40+) httpd processes running. FOGScheduler service is off.

Checking /var/log/httpd/access_log shows the following connections from multiple clients

10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/snapins.checkin.php?mac=D4:3D:7E:25:49:7F%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 4 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/Printers.php?mac=D4:3D:7E:38:FE:60%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 12 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/greenfog.php?mac=74:27:EA:AD:AC:0A%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 73 "-" "-"
10.***.***.***- - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/autologout.php?mac=A4:1F:72:86:3A:A4%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/autologout.php?mac=74:27:EA:CF:9A:FA%7C%7C00:00:00:00:00:00:00:E0%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/servicemodule-active.php?moduleid=taskreboot&mac=A4:1F:72:86:5C:CB%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 5 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:51 +0000] "GET /fog/service/autologout.php?mac=A4:1F:72:86:5C:CB%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:53 +0000] "GET /fog/status/bandwidth.php?dev=eth0 HTTP/1.1" 200 47 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:51 +0000] "GET /fog/service/printerlisting.php?newService=1 HTTP/1.1" 200 5 "-" "-"

And in /var/log/httpd/error_log seeing this

[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::real_connect(): (08004/1040): Too many connections in /var/www/html/fog/lib/db/mysql.class.php on line 30
[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::set_charset(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 31
[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 52
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::real_connect(): (08004/1040): Too many connections in /var/www/html/fog/lib/db/mysql.class.php on line 30
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::set_charset(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 31
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 52
df: `/home/fog/images': Permission denied
df: no file systems processed
[Fri Mar 11 13:22:15 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
[Fri Mar 11 13:22:17 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
[Fri Mar 11 13:22:19 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
[Fri Mar 11 13:22:21 2016] [error] child process 2991 still did not exit, sending a SIGKILL
[Fri Mar 11 13:22:22 2016] [notice] caught SIGTERM, shutting down
[Fri Mar 11 13:23:30 2016] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Mar 11 13:23:30 2016] [notice] Digest: generating secret for digest authentication ...
[Fri Mar 11 13:23:30 2016] [notice] Digest: done
[Fri Mar 11 13:23:31 2016] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.6.19 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
df: `/home/fog/images': Permission denied
df: no file systems processed

From another post I’ve looked at max connections in mysql (this is a CentOS 6.7 server)

su mysql
ulimit -n
1024

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7414
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Does this look ok? Should I look to increase the max connections above 1024?

regards Kiweegie.

Kiweegie

Possible red-herring but worthwhile raising the question I think.

watching connections via

ss -tp

throws up amongst a host of other connections

CLOSE-WAIT 1      0           Fog-server-IP:45260        54.93.153.156:http
CLOSE-WAIT 1      0           Fog-server-IP:45856        54.93.153.156:http

C:\Windows\System32>nslookup 54.93.153.156
Server:  my_AD_Domain controller
Address:   my_AD_Domain controller_IP

Name:    ec2-54-93-153-156.eu-central-1.compute.amazonaws.com
Address:  54.93.153.156

for what reason would FOG be looking to connect to an Amazon IP??? I’m seeing a similar connection attempt to 52.28.249.93 which is another Amazon IP.

There are a load of ESTABLISHED connections showing like below which I believe to be the actual httpd processes though.

Proto Recv-Q Send-Q Local Address                Foreign Address             State
tcp        0      0 ::ffff:Fof_Server_IP:80     ::ffff:Client-IP:64981 ESTABLISHED

If you need any other logs etc to help get to the bottom of this I’m more than happy to oblige.

cheers Kiweegie.

Tom Elliott

@Kiweegie is your fog server reachable from the Internet?

Kiweegie

@Tom-Elliott Nope, not without vpn to our corporate network

Tom Elliott

@Kiweegie There is a couple things the reach out to the internet, one is the FOG Services (Scheduler, MulticastManager, etc…) that reach out to the internet to get the system’s external address in the case that’s the way your node is setup to be reached, so I doubt those are the “problem” things.

Wayne Workman

@Kiweegie Turn down your client checkin time. At work, I’ve got mine set to 300 seconds. IMHO, 60 seconds is absurd.

george1421

FWIW, I have mine set to 900 seconds. I don’t deploy snapins post deployment or issue unattended deployments. There is no real value (in my environment) to have a client check-ins faster than every 15 minutes.

Kiweegie

@Tom-Elliott Morning Tom not sure if related to changes you made for me late last week or a new issue but since the teamviewer session we had, deployment of images fails with error below:

failed to create deployment tasking....

The offending line in path /var/www/html/fog/lib/fog/fogftpclass.php is on line 104

      if (@ftp_login(self::$link,$username,$password) === false) self::ftperror();

no credentials have changed at all. Snapins deploy fine, only images are affected and we have tried with multiple images.

regards, Kiweegie.

Tom Elliott

@Kiweegie I’m not able to replicate this. If snapins are working off of the same nodes that are not sending the image, then is something potentially wrong with how the images are mounted on that system? I know it’s stating an FTP error, so it’s a wee bit odd.

Kiweegie

@Wayne-Workman @george1421 Hi Wayne/George

Where do I find the client checkin time setting? I’ve checked fog settings and nothing shows up there. Could you point me in the right direction please?

cheers Kiweegie

Tom Elliott

@Kiweegie FOG Configuration Page->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME,FOG_GRACE_TIMEOUT

george1421

@Kiweegie FOG Configuration->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME

Kiweegie

@george1421 said:

FOG Configuration->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME

cheers George!

Kiweegie

@Tom-Elliott Hi Tom unfortunately I was going off the information passed to me by my service desk… which turns out to not be as accurate as I thought

I’ve tried deploying snapin and while there is no error thrown up in the GUI, c:\fog.log on target machine shows the same error as seen when trying to deploy image.

 14/03/2016 13:35 Middleware::Communication Unknown Response: Type: 2, File: /var/www/html/fog/lib/fog/fogftp.class.php, Line: 104, Message: ftp_login(): Login incorrect.

This would suggest that the fog or ftp user password has been changed but it has not been altered at least not by me. Entry in following 2 locations matched what I had in keepass

Storage Management > All Storage nodes > DefaultMember > Management Password
FOG Configuration > FOG Settings > TFTP Server > FOG_TFTP_FTP_Password

However, resetting the fog user password at commandline fixed the issue…

Don’t understand who or what amended that password as only I have access to the box and it sure wasn’t me!

cheers . Kiweegie.

Wayne Workman

@Kiweegie The installer a week to two weeks ago set it wrong if there were certain special characters present in the password. It’s since been fixed though. You should verify that what is inside of .fogsettings is correct, and what shows up for the user/pass in the storage management area is correct, and for the kernel update stuff too. All of this is outlined in the “Troubleshoot FTP” wiki article.

Sebastian Roth

@Kiweegie I vaguely remember @Wayne-Workman having seen a fog user password change after running the installer script at some point. Maybe this is related? https://forums.fogproject.org/topic/6649/storage-node-installation-resets-local-fog-user-s-password-to-something-wrong

Cannot actually find the commit that was made to fix this but from what Tom says I thought this was fixed somewhere around version 63xx. Maybe not? Please keep an eye on this and let us know!

Kiweegie

@Wayne-Workman HI Wayne thanks for the heads up. fog password in this case is purely alphanumeric though so don’t think that was it. Resetting the fog password has resolved my issue however so happy to have this one marked as solved.

regards Kiweegie.

High CPU usage from multiple httpd processes version 5800

151

12.0k

17.3k

155.2k