HIGH CPU Fog Services after update r5029 v6759

[Wayne Workman]

@Raymond-Bell Adjust the fields I pointed out earlier, then I’d say reboot.

[Tom Elliott]

@Raymond-Bell For now, I’d say leave the services off until you are sure the load has finally balanced out.

Then start the services one at a time and watch closely to the load. I don’t think starting them, now, would be a problem, but i’m starting to think you’re seeing conflicting runs due to potentially dual webroot’s acting in place. (Again it’s all theory as I really don’t know.)
If you wait five minutes between starting the services, it should give you a good base line as to which one (or multiple) are causing issues.

If you see one service starting to tack up the load, stop it, and go on to the next. If all services do the same we have a baseline to try finding info for/on.

[Raymond Bell]

@Tom-Elliott restarted all services like you suggested and now none of the are spiked, but i still see alot of apache2 processes is this normal in this update?

[baggar11]

I can confirm the 100% cpu usage issue. I don’t have any storage nodes in my setup and have a c2750 based atom processor running my FOG virtual machine. I only have 2 test images, each around 1G.

I found that when I downgraded to version 6753, the high cpu usage disappeared. 6755, 6757 and up all produced the issue.

The next git commit has this in the log.

Author: Tom Elliott <tommygunsster@gmail.com>
Date:   Wed Mar 16 00:42:03 2016 +0000

Ensure variables are set even on initial startup (init.php).

git-svn-id: https://svn.code.sf.net/p/freeghost/code/trunk@5027 71f96598-fa45-0410-b640-bcd6f8691b32

Hope that helps!

-baggar11

[Tom Elliott]

@baggar11 what os is fog running on your server?

[Tom Elliott]

@Raymond-Bell is server load better?

[Raymond Bell]

@Tom-Elliott said:

@Raymond-Bell is server load better?

Yes on server but did the same thing on nodes and they all run HIGH CPU

[baggar11]

@Tom-Elliott

Ubuntu 14.04 here

[Raymond Bell]

@baggar11 said:

@Tom-Elliott

Ubuntu 14.04 here

Same for me Tom

[Wayne Workman]

I just updated my home FOG setup (which includes many nodes) to r6769 and I cannot replicate the issue. I’m using Fedora 23

And my replication setting is set to 60 seconds, and I’ve got the slowest setup in town (running 4 OSs on a single Core 2 duo, and P4s with 100 meg switches).

It’s either a Ubuntu thing or a New Client related thing. And I’m leaning towards it being a Ubuntu thing.

It’s also possible that there is some certain scenario that happened to cause replication to go awol, but we won’t know until we can see a setup that is affected and figure out what’s going on.

[Wayne Workman]

I was just able to test my 1 fog client system at home. it doesn’t make any difference.

I think this is a Ubuntu issue.

[Tom Elliott]

@Wayne-Workman And all, I made a rather significant change (though outside it shouldn’t matter), to hopefully make an attempt at figuring this out.

Basically please try out the latest. First thing I noticed was a very similar result on my storage nodes, (one that is Ubuntu 15, and the other that is Fedora 23) and I found that my particular issue was due to the service sleep time being parsed as a string rather than an integer. This would cause the FOG Services to keep cycling (after initial reboot) probably due to improper connection finding. I’m hoping this is fixed but also a much more performance enhanced FOG server capability.

[baggar11]

@Tom-Elliott Thanks Tom. Have these changes been pushed to Git too? That’s what I’m using…

[Tom Elliott]

@baggar11 Whenever I push, I automatically push to both svn and git. I’m not expecting any miracles though, but would be nice to know if I’m at least kind of on the right track.

[baggar11]

I’m still seeing the issue on the newly pushed 6775. ImageReplicator, MulticastManager and SnapinReplicator seem to be taking all of the cpu load at around 31.6%

ps aux | grep FOG
root       814 31.6  2.3 196196 23976 ?        R    15:39   0:44 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FO ImageReplicator
root       846 31.6  2.3 196244 24116 ?        R    15:39   0:44 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
root       869 31.5  2.3 196196 24068 ?        R    15:39   0:44 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
root       916  0.0  1.8 274228 19244 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FO ImageReplicator
root       918  0.0  1.9 274404 19344 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
root       939  0.0  1.8 274096 19172 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
root      1001  0.1  2.3 196200 24092 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
root      1031  0.1  2.3 196200 24136 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
root      1127  0.0  1.9 274696 19620 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
root      1150  0.0  1.9 274436 19652 ?        S    15:39   0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts

[Wayne Workman]

@baggar11 see how the G is missing?

/opt/fog/service/FOGImageReplicator/FO ImageReplicator

Wonder if that is significant…

[baggar11]

@Wayne-Workman My screen wrapped in my terminal. The “G” is in fact there when I full screen and grep the ps output again.

[Wayne Workman]

@baggar11 Can you give us the last 500 or so Image Replicator logs? FOG Configuration -> LOG Viewer -> Image Replicator.

Also, you might just look through all the logs in there for anything out of place.

[Tom Elliott]

@baggar11 does ubuntu 14.04 use systemctl?

Is there anything spewing into your /etc/apache2/error.log

[Wayne Workman]

@Tom-Elliott said:

@baggar11 does ubuntu 14.04 use systemctl?

No, it does not.