MYSQL/HTTPD resource issues in 5020

[Adam Taylor]

3306 is all localhost connections.

80 is all different clients from around the campus.

[Adam Taylor]

I changed the DB connect from persistant to non persistant to test and see if the connections would then go away.

The connections go up much faster then before (what i expected since no reuse) but once again…they never go down.

That’s all i know to look at/try at the moment. Do appreciate your help in this!

Thanks!

Adam

[Tom Elliott]

@Adam-Taylor I know I keep asking the same things, but please reupdate and try again?

The only other thing that I think could be causing the issue is in line and hopefully this is corrected for. I have no real clue whether or not it is the case though.

[Adam Taylor]

The system is still getting hammered and all four cores are at 100% after a complete reboot.

Adam

[Adam Taylor]

Also, that 100% is always ~30% for mysql and the rest is chewed up with httpd processes.

Adam

[Wayne Workman]

@Adam-Taylor Tom has made some changes, update again.

[Trevelyan]

Changes don’t seem to have fixed anything for me. Still see 100% usage after restart and update to 5078- basically the same as my screenshot earlier.

I do see this in the apache error log though, for all the clients that are trying to connect. Not sure itd make any difference though.
:

[Mon Oct 26 14:55:09.526015 2015] [:error] [pid 3099] [client 10.240.5.151:57133] PHP Warning:  mysqli::mysqli(): (HY000/2002): No such file or directory in /var/www/html/fog/lib/db/MySQL.class.php on line 18

[Tom Elliott]

@Trevelyan this makes absolutely no sense. It sounds like, from the error you are reporting, your server doesn’t have the mysqlnd driver stack for php as it can’t even find the mysqli object.

[Adam Taylor]

I am still seeing 100% usage for me to after the last update with mysql using~30% and httpd using the rest with LOTS of processes.

Adam

[Tom Elliott]

@Adam-Taylor WHen I get back to a normal schedule at work, I’d love to link up and run a Teamviewer connection with you.

It seems, to me, that this isn’t something I’ve been causing, directly. For some time I had some issues, but I have a person who has around 6k hosts and was seeing the resource problems as described. I have confirmed with him that this is now fixed. Now that doesn’t mean an issue still doesn’t persist, but I’m pretty sure this isn’t something FOG is doing any more.

[Adam Taylor]

Can you ask him what OS, apache version, mysql version he is running for a can compare?

Thanks,

Adam

[Adam Taylor]

Question.

Would the old fog client be causing this? We noticed the new MSI fog client but we are still running the old one.

Adam

[Trevelyan]

I’ve noticed that first thing in the morning, our server is fine. Its only when PCs start powering up that we see issues - but it isn’t even a hundred that cause it. Just loads and loads of high-CPU hogging apache2 instances that never seem to end.

For the record - all of these are new clients (we have no PCs running the old client anymore)

[Sebastian Roth]

Are you both running the latest SVN? Would be very interesting to see what is going on within those lots of processes! Maybe you could try using strace (option -p to attach to a running process, -o to write to a file) to see what is going on. I am aware of this being an advanced debugging issue and I am not sure if you are keen enough to get into this. Maybe give it a try and well see what we get. I am more than happy to take a look if you upload a strace log file.

[d4rk3]

I am also experiencing this. This includes random revisions ranging from r4982 - r5080.

[Joseph Hales]

I am also still seeing this on this mornings SVN 5082 on ubuntu 14.

It does not affect the storage node whose cpu load is around 1. Here is TOP of the main server.

lsof -i :80 output
lsof output.txt

[Joseph Hales]

Should this thread be marked solved or should I start a new thread.

[Sebastian Roth]

Maybe it is a good idea to start a (combined) new thread on this issue. Tom and I have been thinking and trying a lot but are unable to reproduce this issue. Probably because we only have a couple to a few dozens of clients to test with. Unfortunately I think we need you - @Adam-Taylor, @Joseph-Hales and @Raymond-Bell - to get into the details of debugging this. I am still not quite sure how we are going to get there. Here are some ideas:

• Check the output of apache2 -V or httpd -V for “Server MPM”, is it Worker or Prefork??
• Apache server status http://fog-ip/server-status (might be enabled already but only accessable from localhost - check out the apache mod-configs)
• Mysql logging, a good talk about this: http://stackoverflow.com/questions/568564/how-can-i-view-live-mysql-queries
• mysqladmin -u root -p -i 1 processlist;
• Tools like mytop and mtop (http://www.tecmint.com/mysql-performance-monitoring/)

Just a few ideas. Probably best to start a new thread and get into the details with this.

[Joseph Hales]

@Uncle-Frank said:

mysqladmin -u root -p -i 1 processlist;

New thread started here
https://forums.fogproject.org/topic/6020/fog-svn-5020-and-above-cpu-hammered-thread/1