MYSQL/HTTPD resource issues in 5020



  • Just updated to svn 5020 and we are having a system resource overload. Mysql doesn’t seem to be dropping any connections now and apache has over 120+ children spawning when then computers check in with the client. From the time I started this topic to right now the sql connections have jumped from 300 to 1100…

    mysql is running around 30-35% CPU and the httpd processes are chewing up the rest of the CPU.

    This only started happening when we updated to 5020 from 4960 (but svn shows revision 4189?)


  • Testers

    @Uncle-Frank said:

    mysqladmin -u root -p -i 1 processlist;

    New thread started here
    https://forums.fogproject.org/topic/6020/fog-svn-5020-and-above-cpu-hammered-thread/1


  • Developer

    Maybe it is a good idea to start a (combined) new thread on this issue. Tom and I have been thinking and trying a lot but are unable to reproduce this issue. Probably because we only have a couple to a few dozens of clients to test with. Unfortunately I think we need you - @Adam-Taylor, @Joseph-Hales and @Raymond-Bell - to get into the details of debugging this. I am still not quite sure how we are going to get there. Here are some ideas:

    Just a few ideas. Probably best to start a new thread and get into the details with this.


  • Testers

    Should this thread be marked solved or should I start a new thread.


  • Testers

    I am also still seeing this on this mornings SVN 5082 on ubuntu 14.
    upload-d6018b50-572b-4c50-9f40-2d700c8b102e

    It does not affect the storage node whose cpu load is around 1. Here is TOP of the main server.
    upload-6f2fe626-bc01-4e99-bd29-8e863a072721

    lsof -i :80 output
    lsof output.txt



  • I am also experiencing this. This includes random revisions ranging from r4982 - r5080.


  • Developer

    Are you both running the latest SVN? Would be very interesting to see what is going on within those lots of processes! Maybe you could try using strace (option -p to attach to a running process, -o to write to a file) to see what is going on. I am aware of this being an advanced debugging issue and I am not sure if you are keen enough to get into this. Maybe give it a try and well see what we get. I am more than happy to take a look if you upload a strace log file.



  • I’ve noticed that first thing in the morning, our server is fine. Its only when PCs start powering up that we see issues - but it isn’t even a hundred that cause it. Just loads and loads of high-CPU hogging apache2 instances that never seem to end.

    For the record - all of these are new clients (we have no PCs running the old client anymore)



  • Question.

    Would the old fog client be causing this? We noticed the new MSI fog client but we are still running the old one.

    Adam



  • Can you ask him what OS, apache version, mysql version he is running for a can compare?

    Thanks,

    Adam


  • Senior Developer

    @Adam-Taylor WHen I get back to a normal schedule at work, I’d love to link up and run a Teamviewer connection with you.

    It seems, to me, that this isn’t something I’ve been causing, directly. For some time I had some issues, but I have a person who has around 6k hosts and was seeing the resource problems as described. I have confirmed with him that this is now fixed. Now that doesn’t mean an issue still doesn’t persist, but I’m pretty sure this isn’t something FOG is doing any more.



  • I am still seeing 100% usage for me to after the last update with mysql using~30% and httpd using the rest with LOTS of processes.

    Adam


  • Senior Developer

    @Trevelyan this makes absolutely no sense. It sounds like, from the error you are reporting, your server doesn’t have the mysqlnd driver stack for php as it can’t even find the mysqli object.



  • Changes don’t seem to have fixed anything for me. Still see 100% usage after restart and update to 5078- basically the same as my screenshot earlier.

    I do see this in the apache error log though, for all the clients that are trying to connect. Not sure itd make any difference though.
    :

    [Mon Oct 26 14:55:09.526015 2015] [:error] [pid 3099] [client 10.240.5.151:57133] PHP Warning:  mysqli::mysqli(): (HY000/2002): No such file or directory in /var/www/html/fog/lib/db/MySQL.class.php on line 18
    


  • @Adam-Taylor Tom has made some changes, update again.



  • Also, that 100% is always ~30% for mysql and the rest is chewed up with httpd processes.

    Adam



  • The system is still getting hammered and all four cores are at 100% after a complete reboot.

    Adam


  • Senior Developer

    @Adam-Taylor I know I keep asking the same things, but please reupdate and try again?

    The only other thing that I think could be causing the issue is in line and hopefully this is corrected for. I have no real clue whether or not it is the case though.



  • I changed the DB connect from persistant to non persistant to test and see if the connections would then go away.

    The connections go up much faster then before (what i expected since no reuse) but once again…they never go down.

    That’s all i know to look at/try at the moment. Do appreciate your help in this!

    Thanks!

    Adam



  • 3306 is all localhost connections.

    80 is all different clients from around the campus.


 

352
Online

41.8k
Users

12.3k
Topics

116.0k
Posts