CPU Usage 100%



  • Server
    • FOG Version: 1.4.3
    • OS: CentOS 6.7
    Client
    • Service Version:
    • OS:
    Description

    For some reason CPU Usage is pretty much always at 100% in vSphere and when I check the top command I see all these httpd processes. Is this because of all the clients that have the FOG client installed?

    It’s currently using a lot of CPU and it really shouldn’t as we use it daily but not that much to justify the amount of CPU.

    Here is a snapshot of the top command:

    0_1503508464796_ea753178-3e90-4faa-a866-b610772f929d-image.png

    And here is a snapshot of vSphere:

    0_1503508546817_ff08925f-57b6-4f57-9149-b834df65bdfa-image.png

    0_1503508572306_b9cb365b-9abd-40b8-8e15-4cb936295c23-image.png

    Any idea what is going on?

    Thanks



  • @george1421 already a problem, i don’t have /etc/httpd/conf.modules.d i have /etc/httpd/modules full of mod_***.so files but none with 00-mpm.conf file :/ could this be a CentOS 6.7 to 7 difference?



  • @george1421 hey yeah it’s still set to 600! should i just follow: https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast instructions?


  • Moderator

    @anthony-delarosa Yes I’m available. Make sure you have a good baseline of usage. Your vmware trending is good enough. If the process works correctly you should not see all of those httpd instances in top since it should spawn php-fpm instances.

    Is your check in time still set to FOG defaults?



  • @george1421 hey! i’m ready to do this today, are you around?



  • @wayne-workman I just noticed that too. @anthony-delarosa I agree, unless there’s a specific reason that you have, I would look into not doing raw.



  • @anthony-delarosa said in CPU Usage 100%:

    the guys have their reasons so they do a raw windows 7, 8.1, 10 images

    This is off topic I know, but you should really have them use regular images instead of raw - there’s several options - ones that resize, ones that do not, lots of compression options. It’s light-speeds faster than raw, I really do mean light-speeds faster.



  • @anthony-delarosa That’s nuts. Glad to see it in such a large deployment. Have you thought about doing golden images and just having varied images for software deployments?



  • @THEMCV both different in software and different model plus different OS so yeah it adds up! Mind you we’re a company with over 700 fx/animators/lighters/compositors etc…



  • @anthony-delarosa Wow, that is a lot. Consider me in awe of your installation. : )

    So is each image different software or is it more for each model?



  • @themcv glad to know we are the first then :D we use FOG for all our images and since I work for a studios company with various different shows, each show kinda has their own image! the guys have their reasons so they do a raw windows 7, 8.1, 10 images for all models then a pre-sys image once they have all the applications installed and then a production image so times 3x images per OS per model! yeah that adds up quickly lol



  • @anthony-delarosa Do you have golden images going? I’ve not ever heard of 2TB of images before! Then again, I don’t think anybody has had as big of a FOG environment as you. : )


  • Moderator

    @anthony-delarosa Ok when you are ready we can move forward. These instuctions are not destructive and can be reversed with a comment in a config file and restarting apache.

    Here is where I’ve been documenting the config changes: https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast

    The change to my server has made a difference in web gui response time. But again I don’t have a large campus so its hard to scale how much of a change its made. The instructions I created were for centos 7, but should translate to 6.7 without issue. If you run into an issue I’m spinning up a 6.7 instance of centos just to be sure.

    As part of your task. I would like you to track in vmware vm cpu usage to see when you turn on php-fpm if there is a change (in either direction) or no change. There are two parts I want to test one step at a time.

    1. Enable php-fpm
    2. Turn on memcache to store temporary session information in memory instead of the file system.

    Your vm should have 3-4 GB of ram for this test. If you have more already then that is fine. I just want to make sure we have room to spawn up everything.



  • hey @george1421 how ya doing? yeah sure I can help you guys out! can we schedule this for next week? had a backup meltdown n FOG hadn’t been backed up in a few days so Veeam is doing it’s FULL BACKUP and since our FOG server has over 2 TBs of images it’s backing that up now but should be done soon. Next week works?


  • Moderator

    If you have time to help us test this new configuration, please let me know and I will provide instructions.


  • Moderator

    @anthony-delarosa Yes that is the right setting, default check in time is 5 minutes. But if you can hold off a day or so until we can test the php-fpm code. I want to see if it makes a difference on the default settings for FOG. In ‘theory’ it should make a dramatic difference in overall FOG performance.


  • Moderator

    @anthony-delarosa Excellent, thank you.

    Here are my thoughts on how to set this up. I’m only posting below for my reference. I want to spin up a Cento 6.7 FOG server tomorrow and test the process out before you implement it. I also want to see if memcache will help with the process too. But I need to do a bit more research first.

    # yum install php-fpm
    # chkconfig --levels 235 php-fpm on
    
    Edit /etc/php-fpm.d/www.conf to use sockets
    
    # vi /etc/php-fpm.d/www.conf
    
    ;listen = 127.0.0.1:9000
    listen = /tmp/php5-fpm.sock
    
    listen.owner = apache
    listen.group = apache
    
    # service php-fpm start
    
    Setting up Apache and mod_fastcgi
    
    Install the RPMForge repo:
    
    # wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
    # rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
    
    Add some priorities which repo to use:
    
    # yum install yum-priorities
     
    # vi /etc/yum.repos.d/epel.repo 
    ... add the line priority=10 to the [epel] section
    
    # yum install mod_fastcgi
    
    Configure mod_fastcgi
    
    If you have php enabled disable it
    
    # mv /etc/httpd/conf.d/{php.conf,php.conf.disable}
    
    # mkdir /usr/lib/cgi-bin/
    
    # vi /etc/httpd/conf.d/mod_fastcgi.conf
    
    LoadModule fastcgi_module modules/mod_fastcgi.so
     
    <IfModule mod_fastcgi.c>
    	DirectoryIndex index.php index.html index.shtml index.cgi
    	AddHandler php5-fcgi .php
    	Action php5-fcgi /php5-fcgi
    	Alias /php5-fcgi /usr/lib/cgi-bin/php5-fcgi
    	FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -socket /tmp/php5-fpm.sock -pass-header Authorization
     
    	# For monitoring status with e.g. Munin
    	<LocationMatch "/(ping|status)">
    		SetHandler php5-fcgi-virt
    		Action php5-fcgi-virt /php5-fcgi virtual
    	</LocationMatch>
    </IfModule>
    
    # service httpd restart
    
    


  • @george1421 is this the setting i can increase? the FOG_CHECKIN_TIMEOUT ?

    0_1503535835621_c282f422-2be6-49a6-a1c8-cc2feb0fe52a-image.png



  • @george1421 hey George! yeah i think i can help you out there! send me the detailed steps i need to take and i can always take a snapshot of this so if all fails i can revert back!



  • @anthony-delarosa said in CPU Usage 100%:

    @george1421 we do, easily over 800 clients! i was not aware that the clients would be checking in with FOG! I thought it was FOG that would contact the host if it needed to communicate something to it

    The fog client uses polling to communicate with the fog server, this is detailed here: https://wiki.fogproject.org/wiki/index.php?title=FOG_Client#Polling_Behavior


 

389
Online

41.4k
Users

11.8k
Topics

112.1k
Posts