HIGH CPU Fog Services after update r5029 v6759

[Tom Elliott]

@baggar11 what os is fog running on your server?

[Tom Elliott]

@Raymond-Bell is server load better?

[Raymond Bell]

@Tom-Elliott said:

@Raymond-Bell is server load better?

Yes on server but did the same thing on nodes and they all run HIGH CPU

[baggar11]

@Tom-Elliott

Ubuntu 14.04 here

[Raymond Bell]

@baggar11 said:

@Tom-Elliott

Ubuntu 14.04 here

Same for me Tom

[Wayne Workman]

I just updated my home FOG setup (which includes many nodes) to r6769 and I cannot replicate the issue. I’m using Fedora 23

And my replication setting is set to 60 seconds, and I’ve got the slowest setup in town (running 4 OSs on a single Core 2 duo, and P4s with 100 meg switches).

It’s either a Ubuntu thing or a New Client related thing. And I’m leaning towards it being a Ubuntu thing.

It’s also possible that there is some certain scenario that happened to cause replication to go awol, but we won’t know until we can see a setup that is affected and figure out what’s going on.

[Wayne Workman]

I was just able to test my 1 fog client system at home. it doesn’t make any difference.

I think this is a Ubuntu issue.

[Tom Elliott]

@Wayne-Workman And all, I made a rather significant change (though outside it shouldn’t matter), to hopefully make an attempt at figuring this out.

Basically please try out the latest. First thing I noticed was a very similar result on my storage nodes, (one that is Ubuntu 15, and the other that is Fedora 23) and I found that my particular issue was due to the service sleep time being parsed as a string rather than an integer. This would cause the FOG Services to keep cycling (after initial reboot) probably due to improper connection finding. I’m hoping this is fixed but also a much more performance enhanced FOG server capability.

[baggar11]

@Tom-Elliott Thanks Tom. Have these changes been pushed to Git too? That’s what I’m using…

[Tom Elliott]

@baggar11 Whenever I push, I automatically push to both svn and git. I’m not expecting any miracles though, but would be nice to know if I’m at least kind of on the right track.

[baggar11]

I’m still seeing the issue on the newly pushed 6775. ImageReplicator, MulticastManager and SnapinReplicator seem to be taking all of the cpu load at around 31.6%

ps aux | grep FOG
root       814 31.6  2.3 196196 23976 ?        R    15:39   0:44 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FO ImageReplicator
root       846 31.6  2.3 196244 24116 ?        R    15:39   0:44 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
root       869 31.5  2.3 196196 24068 ?        R    15:39   0:44 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
root       916  0.0  1.8 274228 19244 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FO ImageReplicator
root       918  0.0  1.9 274404 19344 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
root       939  0.0  1.8 274096 19172 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
root      1001  0.1  2.3 196200 24092 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
root      1031  0.1  2.3 196200 24136 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
root      1127  0.0  1.9 274696 19620 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
root      1150  0.0  1.9 274436 19652 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts

[Wayne Workman]

@baggar11 see how the G is missing?

/opt/fog/service/FOGImageReplicator/FO ImageReplicator

Wonder if that is significant…

[baggar11]

@Wayne-Workman My screen wrapped in my terminal. The “G” is in fact there when I full screen and grep the ps output again.

[Wayne Workman]

@baggar11 Can you give us the last 500 or so Image Replicator logs? FOG Configuration -> LOG Viewer -> Image Replicator.

Also, you might just look through all the logs in there for anything out of place.

[Tom Elliott]

@baggar11 does ubuntu 14.04 use systemctl?

Is there anything spewing into your /etc/apache2/error.log

[Wayne Workman]

@Tom-Elliott said:

@baggar11 does ubuntu 14.04 use systemctl?

No, it does not.

[baggar11]

@Wayne-Workman Here you go. 500 lines of logs.

EDIT: Forgot to select Image Replicator. Here it is.

https://www.refheap.com/116045

[baggar11]

@Tom-Elliott Here’s what I’m seeing after stopping all services, clearing the logs and then rebooting.

cat error.log 
[Wed Mar 16 16:10:53.258602 2016] [mpm_prefork:notice] [pid 946] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g configured -- resuming normal operations
[Wed Mar 16 16:10:53.258795 2016] [core:notice] [pid 946] AH00094: Command line: '/usr/sbin/apache2'
[Wed Mar 16 16:11:35.735081 2016] [:error] [pid 950] [client 192.168.10.15:53300] PHP Strict Standards:  Only variables should be passed by reference in /var/www/fog/lib/pages/dashboardpage.class.php on line 71, referer: http://192.168.10.14/fog/management/index.php

[Tom Elliott]

@baggar11 What services are using the bulk of the CPU?

[baggar11]

@Tom-Elliott For a full view of processes, I gave an output of ps aux | grep FOG down below. These are the top 3 though.

FOGImageReplicator
FOGMulticastManager
FOGSnapinReplicator