High CPU usage from multiple httpd processes version 5800

[Kiweegie]

@Tom-Elliott Sorry got sidetracked by lunch

Still several (40+) httpd processes running. FOGScheduler service is off.

Checking /var/log/httpd/access_log shows the following connections from multiple clients

10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/snapins.checkin.php?mac=D4:3D:7E:25:49:7F%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 4 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/Printers.php?mac=D4:3D:7E:38:FE:60%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 12 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/greenfog.php?mac=74:27:EA:AD:AC:0A%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 73 "-" "-"
10.***.***.***- - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/autologout.php?mac=A4:1F:72:86:3A:A4%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/autologout.php?mac=74:27:EA:CF:9A:FA%7C%7C00:00:00:00:00:00:00:E0%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/servicemodule-active.php?moduleid=taskreboot&mac=A4:1F:72:86:5C:CB%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 5 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:51 +0000] "GET /fog/service/autologout.php?mac=A4:1F:72:86:5C:CB%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:53 +0000] "GET /fog/status/bandwidth.php?dev=eth0 HTTP/1.1" 200 47 "-" "-"
10.***.***.*** - - [11/Mar/2016:14:26:51 +0000] "GET /fog/service/printerlisting.php?newService=1 HTTP/1.1" 200 5 "-" "-"

And in /var/log/httpd/error_log seeing this

[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::real_connect(): (08004/1040): Too many connections in /var/www/html/fog/lib/db/mysql.class.php on line 30
[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::set_charset(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 31
[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
[Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 52
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::real_connect(): (08004/1040): Too many connections in /var/www/html/fog/lib/db/mysql.class.php on line 30
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::set_charset(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 31
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
[Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 52
df: `/home/fog/images': Permission denied
df: no file systems processed
[Fri Mar 11 13:22:15 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
[Fri Mar 11 13:22:17 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
[Fri Mar 11 13:22:19 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
[Fri Mar 11 13:22:21 2016] [error] child process 2991 still did not exit, sending a SIGKILL
[Fri Mar 11 13:22:22 2016] [notice] caught SIGTERM, shutting down
[Fri Mar 11 13:23:30 2016] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Mar 11 13:23:30 2016] [notice] Digest: generating secret for digest authentication ...
[Fri Mar 11 13:23:30 2016] [notice] Digest: done
[Fri Mar 11 13:23:31 2016] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.6.19 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
df: `/home/fog/images': Permission denied
df: no file systems processed

From another post I’ve looked at max connections in mysql (this is a CentOS 6.7 server)

su mysql
ulimit -n
1024

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7414
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Does this look ok? Should I look to increase the max connections above 1024?

regards Kiweegie.

[Kiweegie]

Possible red-herring but worthwhile raising the question I think.

watching connections via

ss -tp

throws up amongst a host of other connections

CLOSE-WAIT 1      0           Fog-server-IP:45260        54.93.153.156:http
CLOSE-WAIT 1      0           Fog-server-IP:45856        54.93.153.156:http

C:\Windows\System32>nslookup 54.93.153.156
Server:  my_AD_Domain controller
Address:   my_AD_Domain controller_IP

Name:    ec2-54-93-153-156.eu-central-1.compute.amazonaws.com
Address:  54.93.153.156

for what reason would FOG be looking to connect to an Amazon IP??? I’m seeing a similar connection attempt to 52.28.249.93 which is another Amazon IP.

There are a load of ESTABLISHED connections showing like below which I believe to be the actual httpd processes though.

Proto Recv-Q Send-Q Local Address                Foreign Address             State
tcp        0      0 ::ffff:Fof_Server_IP:80     ::ffff:Client-IP:64981 ESTABLISHED

If you need any other logs etc to help get to the bottom of this I’m more than happy to oblige.

cheers Kiweegie.

[Tom Elliott]

@Kiweegie is your fog server reachable from the Internet?

[Kiweegie]

@Tom-Elliott Nope, not without vpn to our corporate network

[Tom Elliott]

@Kiweegie There is a couple things the reach out to the internet, one is the FOG Services (Scheduler, MulticastManager, etc…) that reach out to the internet to get the system’s external address in the case that’s the way your node is setup to be reached, so I doubt those are the “problem” things.

[Wayne Workman]

@Kiweegie Turn down your client checkin time. At work, I’ve got mine set to 300 seconds. IMHO, 60 seconds is absurd.

[george1421]

FWIW, I have mine set to 900 seconds. I don’t deploy snapins post deployment or issue unattended deployments. There is no real value (in my environment) to have a client check-ins faster than every 15 minutes.

[Kiweegie]

@Tom-Elliott Morning Tom not sure if related to changes you made for me late last week or a new issue but since the teamviewer session we had, deployment of images fails with error below:

The offending line in path /var/www/html/fog/lib/fog/fogftpclass.php is on line 104

      if (@ftp_login(self::$link,$username,$password) === false) self::ftperror();

no credentials have changed at all. Snapins deploy fine, only images are affected and we have tried with multiple images.

regards, Kiweegie.

[Tom Elliott]

@Kiweegie I’m not able to replicate this. If snapins are working off of the same nodes that are not sending the image, then is something potentially wrong with how the images are mounted on that system? I know it’s stating an FTP error, so it’s a wee bit odd.

[Kiweegie]

@Wayne-Workman @george1421 Hi Wayne/George

Where do I find the client checkin time setting? I’ve checked fog settings and nothing shows up there. Could you point me in the right direction please?

cheers Kiweegie

[Tom Elliott]

@Kiweegie FOG Configuration Page->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME,FOG_GRACE_TIMEOUT

[george1421]

@Kiweegie FOG Configuration->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME

[Kiweegie]

@george1421 said:

FOG Configuration->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME

cheers George!

[Kiweegie]

@Tom-Elliott Hi Tom unfortunately I was going off the information passed to me by my service desk… which turns out to not be as accurate as I thought

I’ve tried deploying snapin and while there is no error thrown up in the GUI, c:\fog.log on target machine shows the same error as seen when trying to deploy image.

 14/03/2016 13:35 Middleware::Communication Unknown Response: Type: 2, File: /var/www/html/fog/lib/fog/fogftp.class.php, Line: 104, Message: ftp_login(): Login incorrect.

This would suggest that the fog or ftp user password has been changed but it has not been altered at least not by me. Entry in following 2 locations matched what I had in keepass

Storage Management > All Storage nodes > DefaultMember > Management Password
FOG Configuration > FOG Settings > TFTP Server > FOG_TFTP_FTP_Password

However, resetting the fog user password at commandline fixed the issue…

Don’t understand who or what amended that password as only I have access to the box and it sure wasn’t me!

cheers . Kiweegie.

[Wayne Workman]

@Kiweegie The installer a week to two weeks ago set it wrong if there were certain special characters present in the password. It’s since been fixed though. You should verify that what is inside of .fogsettings is correct, and what shows up for the user/pass in the storage management area is correct, and for the kernel update stuff too. All of this is outlined in the “Troubleshoot FTP” wiki article.

[Sebastian Roth]

@Kiweegie I vaguely remember @Wayne-Workman having seen a fog user password change after running the installer script at some point. Maybe this is related? https://forums.fogproject.org/topic/6649/storage-node-installation-resets-local-fog-user-s-password-to-something-wrong

Cannot actually find the commit that was made to fix this but from what Tom says I thought this was fixed somewhere around version 63xx. Maybe not? Please keep an eye on this and let us know!

[Kiweegie]

@Wayne-Workman HI Wayne thanks for the heads up. fog password in this case is purely alphanumeric though so don’t think that was it. Resetting the fog password has resolved my issue however so happy to have this one marked as solved.

regards Kiweegie.

[Sebastian Roth]

@Wayne-Workman Should have reloaded the page… Good to know that you seam to have verified that this is fixed. Probably it was still wrong in .fogsettings for @Kiweegie?!

[Tom Elliott]

@Sebastian-Roth The real question is the original issue that caused this post to be opened in the first place fixed?

@Kiweegie, are you solving this because the password/snapin issue is fixed or your server is no longer sufferring from the High CPU Usage from multiple httpd processes?