CPU Usage 100%

[anthony.delarosa]

@themcv hey, i upgraded using GIT and now i’m on 1.4.4 and now running top:

and this is vSphere performance for CPU

does FOG generally use that muchCPU MHz? it is the VM that uses most CPU in this VM Host:

Seems like quite a lot to me

[george1421]

@anthony-delarosa How many client computers (with fog client installed) are on your network? With mysql at 27% it sounds like you have quite a few checking in.

[anthony.delarosa]

@george1421 we do, easily over 800 clients! i was not aware that the clients would be checking in with FOG! I thought it was FOG that would contact the host if it needed to communicate something to it

[george1421]

@anthony-delarosa No the fog client works the other way around, the clients check into the master fog server for things to do. They also post inventory info and such.

Since you have a great performance data so far, I wonder if you would be willing to help the fog project by doing something I had on the back burner for a while.

The idea is simply to install php-fpm and have apache use that instead of its internal php engine. I’m interested to know how that impact:

1. Performance of the web gui
2. Impact on performance of the entire fog server CPU usage.

I see you are using an older OS, but the php-fpm package should still be available in epel or in the centos repository. If you have a little time to test this idea let me know and I will put together a tutorial on how to set this up. Reverting back to the default apache php can be done by just commenting out one line. I would do it on my production fog server, but I only have a small campus. I’ve been looking for someone who had around 1000 clients to see if the test was dramatic.

[george1421]

@anthony-delarosa In your case you can in crease the check-in delay to help with cpu usage on your fog server. You can do this with the understanding that it will also take longer for clients to see snapin deployments and other assigned tasks. It will not impact image deployment only fog client tasks. (but I’d like to see the results of the php-fpm test before you make this change if you agree).

[Wayne Workman]

@anthony-delarosa said in CPU Usage 100%:

@george1421 we do, easily over 800 clients! i was not aware that the clients would be checking in with FOG! I thought it was FOG that would contact the host if it needed to communicate something to it

The fog client uses polling to communicate with the fog server, this is detailed here: https://wiki.fogproject.org/wiki/index.php?title=FOG_Client#Polling_Behavior

[anthony.delarosa]

@george1421 hey George! yeah i think i can help you out there! send me the detailed steps i need to take and i can always take a snapshot of this so if all fails i can revert back!

[anthony.delarosa]

@george1421 is this the setting i can increase? the FOG_CHECKIN_TIMEOUT ?

[george1421]

@anthony-delarosa Excellent, thank you.

Here are my thoughts on how to set this up. I’m only posting below for my reference. I want to spin up a Cento 6.7 FOG server tomorrow and test the process out before you implement it. I also want to see if memcache will help with the process too. But I need to do a bit more research first.

# yum install php-fpm
# chkconfig --levels 235 php-fpm on

Edit /etc/php-fpm.d/www.conf to use sockets

# vi /etc/php-fpm.d/www.conf

;listen = 127.0.0.1:9000
listen = /tmp/php5-fpm.sock

listen.owner = apache
listen.group = apache

# service php-fpm start

Setting up Apache and mod_fastcgi

Install the RPMForge repo:

# wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
# rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

Add some priorities which repo to use:

# yum install yum-priorities
 
# vi /etc/yum.repos.d/epel.repo 
... add the line priority=10 to the [epel] section

# yum install mod_fastcgi

Configure mod_fastcgi

If you have php enabled disable it

# mv /etc/httpd/conf.d/{php.conf,php.conf.disable}

# mkdir /usr/lib/cgi-bin/

# vi /etc/httpd/conf.d/mod_fastcgi.conf

LoadModule fastcgi_module modules/mod_fastcgi.so
 
<IfModule mod_fastcgi.c>
	DirectoryIndex index.php index.html index.shtml index.cgi
	AddHandler php5-fcgi .php
	Action php5-fcgi /php5-fcgi
	Alias /php5-fcgi /usr/lib/cgi-bin/php5-fcgi
	FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -socket /tmp/php5-fpm.sock -pass-header Authorization
 
	# For monitoring status with e.g. Munin
	<LocationMatch "/(ping|status)">
		SetHandler php5-fcgi-virt
		Action php5-fcgi-virt /php5-fcgi virtual
	</LocationMatch>
</IfModule>

# service httpd restart

[george1421]

@anthony-delarosa Yes that is the right setting, default check in time is 5 minutes. But if you can hold off a day or so until we can test the php-fpm code. I want to see if it makes a difference on the default settings for FOG. In ‘theory’ it should make a dramatic difference in overall FOG performance.

[george1421]

If you have time to help us test this new configuration, please let me know and I will provide instructions.

[anthony.delarosa]

hey @george1421 how ya doing? yeah sure I can help you guys out! can we schedule this for next week? had a backup meltdown n FOG hadn’t been backed up in a few days so Veeam is doing it’s FULL BACKUP and since our FOG server has over 2 TBs of images it’s backing that up now but should be done soon. Next week works?

[george1421]

@anthony-delarosa Ok when you are ready we can move forward. These instuctions are not destructive and can be reversed with a comment in a config file and restarting apache.

Here is where I’ve been documenting the config changes: https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast

The change to my server has made a difference in web gui response time. But again I don’t have a large campus so its hard to scale how much of a change its made. The instructions I created were for centos 7, but should translate to 6.7 without issue. If you run into an issue I’m spinning up a 6.7 instance of centos just to be sure.

As part of your task. I would like you to track in vmware vm cpu usage to see when you turn on php-fpm if there is a change (in either direction) or no change. There are two parts I want to test one step at a time.

1. Enable php-fpm
2. Turn on memcache to store temporary session information in memory instead of the file system.

Your vm should have 3-4 GB of ram for this test. If you have more already then that is fine. I just want to make sure we have room to spawn up everything.

[THEMCV]

@anthony-delarosa Do you have golden images going? I’ve not ever heard of 2TB of images before! Then again, I don’t think anybody has had as big of a FOG environment as you. : )

[anthony.delarosa]

@themcv glad to know we are the first then we use FOG for all our images and since I work for a studios company with various different shows, each show kinda has their own image! the guys have their reasons so they do a raw windows 7, 8.1, 10 images for all models then a pre-sys image once they have all the applications installed and then a production image so times 3x images per OS per model! yeah that adds up quickly lol

[THEMCV]

@anthony-delarosa Wow, that is a lot. Consider me in awe of your installation. : )

So is each image different software or is it more for each model?

[anthony.delarosa]

@THEMCV both different in software and different model plus different OS so yeah it adds up! Mind you we’re a company with over 700 fx/animators/lighters/compositors etc…

[THEMCV]

@anthony-delarosa That’s nuts. Glad to see it in such a large deployment. Have you thought about doing golden images and just having varied images for software deployments?

[Wayne Workman]

@anthony-delarosa said in CPU Usage 100%:

the guys have their reasons so they do a raw windows 7, 8.1, 10 images

This is off topic I know, but you should really have them use regular images instead of raw - there’s several options - ones that resize, ones that do not, lots of compression options. It’s light-speeds faster than raw, I really do mean light-speeds faster.

[THEMCV]

@wayne-workman I just noticed that too. @anthony-delarosa I agree, unless there’s a specific reason that you have, I would look into not doing raw.