Fog SVN 5020 and above CPU Hammered thread.

[Wayne Workman]

I’m fairly certain that the FOG Dashboard is causing the high CPU usage.

after moving the dashboard’s js file and refreshing the FOG Dashboard, I saw a 0.02% 1-minute CPU Load average while sitting on the FOG Dashboard page.

mv /var/www/html/fog/management/js/fog/fog.dashboard.js /var/www/html/fog/management/js/fog/fog.dashboard.js.moved

After putting it back and refreshing the FOG Dashboard, I saw a 0.24% 1-minute CPU Load average while sitting on the FOG Dashboard page.

mv /var/www/html/fog/management/js/fog/fog.dashboard.js.moved /var/www/html/fog/management/js/fog/fog.dashboard.js

Opening three tabs with the file in the correct place, the 1-minute CPU Load average was at 0.96%

Also, each FOG Dashboard page creates approximately 2 httpd processes. So with 3 pages open and sitting on the dashboard, I was seeing about 6 httpd instances in top. So, my hypothesis is that organizations with multiple FOG users under 1 main server probably have the FOG Dashboard open all the time, causing the load.

I’m not saying they should close the dashboard or not leave it open all day… it’s probably a recent code change somewhere on the dashboard causing this new issue…

SO… if you all could please test, and just temporarily move the file and then tell everybody to refresh their FOG tabs, just see what the CPU does and please report back.

[Sebastian Roth]

@Wayne-Workman Interesting catch we might have to look into as well. On the other hand I am pretty sure this is not causing the high load on those servers with a lot of clients in the environment. On Joseph Hales’ server we see a lot of apache processes spawned and kept open by clients. Those clients never request Javascript files AFAIK.

I will be setting up an older revision with Joseph later today and I really hope to see a difference. We then step up the revisions to see where it all started. Keep your fingers crossed.

[d4rk3]

I know for certain it started somewhere between r4960 and r4982.

[Wayne Workman]

For what it’s worth, there are about 500ish PCs here in my building that are using the legacy client, and I’m using the latest as of yesterday and I’m not seeing the large CPU loads as everyone else is…

Also, nobody here but me and my co-worker even use the fog web interface, and we don’t leave it sitting open either.

[d4rk3]

@Wayne-Workman Even if the web interface isn’t used by anyone the load will always be high.

However, out of the ~1000 clients I have approximately 900 have the new client installed.

Maybe this issue is related to these newest revisions with regard to communication with the new client (v0.95).

[Tom Elliott]

Can all update? My suspicion as to a part of the problem is specific to the fog multicast service. I noticed things were not starting for it and would continuously be respawned. This would create quite a high load on the system as it would be constantly creating new sessions and spawning new instances. While I’m sure there may be some cases of weirdness I think this may be a largely attributing factor.

[Joseph Hales]

I am still seeing the issue after updating and restarting SVN 5106. Also after updating to 5108.

[Trevelyan]

Still seeing the issue too. Note the huge number of threads and tasks!

[Tom Elliott]

Guys,

Do me a favor, it’s going to break things, but it’s very specific.

It’s the service module active checks. They’re spawning way too many processes.

For now, just run: mv /var/www/fog/service/servicemodule-active.php /var/www/fog/service/service-active.php

[Trevelyan]

Well it broke it but it stopped the “thing”!

[Tom Elliott]

@Trevelyan I’m aware of it breaking the client stuff, but it’s all i can do for right now.

I’m remoted into another system having the same type of issues and trying to use it as a way to figure out exactly what’s up. You can image, and do all that, just no FOG Client stuff at the moment. I’ll be trying to narrow exactly where it is.

[Tom Elliott]

While it’s not perfect, I found 4168 is still good and 4169 things are “decent”. 4170, all hell breaks loose. Please update back to 4168 to be “functional”.

You can do this with svn by cd /opt/trunk;svn up -r 4168;cd bin;./installfog.sh

Of course change the /opt/trunk path to the location of your svn folder.

[Adam Taylor]

4168 == sanity!

Never get past about 30% now.

Adam

[Joseph Hales]

Ok im functional at 4168 just a note all the snapins that weren’t showing up are all there now.

[Tom Elliott]

I’m now fairly certain these cpu load issues is now fixed. It was due to HookManager. I’m sorry it took so long to figure out.

[Trevelyan]

Everything seems happy now!

Thanks

PS - Hosts and Users report shows up as error 500 - links to (fog server…) /node=report&sub=file&f=SG9zdHMgYW5kIFVzZXJzLnBocA==

[Tom Elliott]

@Trevelyan I’ll fix the hosts and users report when I get into work.

Can you all verify if things are better and let me know? I’m sorry about the problem this caused, but hey progress or something.

[Sebastian Roth]

I want to thank Tom again for this! I tried to assist and help but I just don’t know the code (and its history) as much as Tom does. He’s done all the hard work to find and fix this issue. Along the way he also fixed a couple other things as well.

@Tom-Elliott Don’t say sorry. FOG is work in progress and you are pushing things way forward!! :metal:

[Mark Buteyn]

@Tom-Elliott Thanks for all your work on fog. Long time user. I’m on 5124 and CPU utilization looks good. I echo @Trevelyan 's concern about the user tracker report. CPU utilization goes high and then comes back down after the 500 error is sent to the browser. I know you said you’d fix it when you got to work. Is there a way for me to find out what version fixes it? Then I won’t bother posting this question in the forum.

Thanks,

Mark

[Joseph Hales]

I updated my main and storage node after the update to SVN 5126 the CPU usage is normal but something is still wrong with mysql the database hits MAX_Workers right away and this prevents the storage node from connecting.

[Thu Oct 29 13:34:44.593182 2015] [:error] [pid 21227] [client 10.109.49.21:54036] PHP Warning:  mysqli::mysqli(): MySQL server has gone away in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:34:44.593262 2015] [:error] [pid 21227] [client 10.109.49.21:54036] PHP Warning:  mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:34:44.681319 2015] [:error] [pid 21145] [client 10.114.51.136:49375] PHP Warning:  mysqli::mysqli(): MySQL server has gone away in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:34:44.681414 2015] [:error] [pid 21145] [client 10.114.51.136:49375] PHP Warning:  mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:34:44.837520 2015] [mpm_prefork:notice] [pid 21097] AH00169: caught SIGTERM, shutting down
[Thu Oct 29 13:36:56.973677 2015] [mpm_prefork:notice] [pid 1510] AH00163: Apache/2.4.16 (Ubuntu) OpenSSL/1.0.1f configured -- resuming normal operations
[Thu Oct 29 13:36:56.981375 2015] [core:notice] [pid 1510] AH00094: Command line: '/usr/sbin/apache2'
[Thu Oct 29 13:38:00.215860 2015] [mpm_prefork:error] [pid 1510] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting```