Ubuntu CPU Usage High After FOG Update
-
Hi All,
I recently updated FOG from 1.4.3 to 1.5.4 and since the update the CPU usage on the Ubuntu (16.04 LTS) VM it is installed on has gone through the roof and it is affecting the performance of FOG and the web interface! It never had any issues before hand so I’m not sure what has gone wrong?
I have tried throwing more resources at it e.g. went from one vCPU to two, and also from 4GB of RAM to 12GB. The extra CPU has improved the responsiveness of the FOG web interface but CPU usage is still way too high it seems.
A restart of the Ubuntu VM has not fixed the issue either. There are no active tasks showing in FOG, and we only use it for imaging devices, no scheduled tasks or anything like that.
I have looked at the ‘top’ command and it appears that “mysql” and “www-data” are using the most.
Any ideas what could be wrong?
Thankyou,
Derek
-
here are a couple of issues that have come up since 1.5.4 has been released that have been fixed in 1.5.5 (still unreleased at this time). But it is a bit strange that you have mysql pulling the most load. Lets start by doing the following to get back to something stable.
- Roll your FOS kernel back to 4.15.2 to get past the mbr/gpt issue. Make sure you download both the 32 and 64 bit PC kernels (not ARM).
- Edit the www.conf file to update/include these settings, then reboot your FOG server. Search for the www.conf file in /etc. Depending on which version of php installed it will be in a different location.
pm.max_children = 35 pm.max_requests = 2000 php_admin_value[memory_limit] = 256Then reboot your FOG server. This will not address the mysql, but it should resolve the webui issues.
How many fog clients communicate with your FOG server?
What is your client check in time for the fog clients?If you are having issues with multicasting then we will need to have you switch to the 1.5.5 working branch to get that update.
-
Thankyou george1421 for your reply!
Do you have any instructions/guidance around rolling back the FOS kernel? I’m not familiar with this process and don’t want to break anything. My knowledge on linux/ubuntu systems is limited as I only use it to run FOG. We just tried to image a laptop and it seems to have got stuck at the GPT section so I’m guessing this is the issue you mentioned?
I have found a “www.conf” file in the following path, would this be the right one to edit?
“/etc/php/7.1/fpm/pool.d/www.conf”As far as clients communicating. We have about 600 devices on the network that could potentially be communicating with the FOG server if they were all turned on at once. I have not had any issues with this in the past.
I am unsure what the client check in time is, where would I find that setting as I have had a bit of a look around and can’t see anything? I don’t believe I’ve ever changed a setting like that so it is still probably the default check in time value.
Thinking about the client though, I have not touched/updated the client on our devices since upgrading to FOG 1.5.4, could this potentially be an issue? I checked one of the “fog.log” files on a client and it looks like it is communicating every couple of minutes according to the time stamps in the log.
-
Hi George1421,
I just found the client check in time setting in the “FOG Settings” page, it is set to “60”.
I was also wondering if you meant “FOG kernel” rather than “FOS”? If so, I believe I understand how to do this by just swapping the “bzImage” and “bzImage32” files out in the following directory?
“/var/www/fog/service/ipxe/”If this is correct, could you just please point me to the best location to source the kernels from these days?
Thankyou,
Derek
-
@derek-newbold said in Ubuntu CPU Usage High After FOG Update:
Hi George1421,
I just found the client check in time setting in the “FOG Settings” page, it is set to “60”.
60 seconds is probably why your mysql process time is pretty high. This mean if you have 100 computers where the fog service is installed you will have all 100 check in every minute. You can probably bump that setting to 600 seconds (10 minutes). This means when you deploy a snapin it could take up to 10 minutes before the client sees this task.
I was also wondering if you meant “FOG kernel” rather than “FOS”? If so, I believe I understand how to do this by just swapping the “bzImage” and “bzImage32” files out in the following directory?
“/var/www/fog/service/ipxe/”The term FOG and FOS kernels are pretty much the same. The orginal name was FOG kernels, but then the devlopers decided to make a distinction (it help with explaining) FOS (Fog OS) is the linux operating system that runs on the target computer that captures and deploys images. FOG now refers to the webgui and server side stuff.
You can downgrade the kernels by hand using the path you defined or just go into the FOG Settings -> FOG Kernels (hint name needs to be changed) And then pick the older kernel 4.15.2 for both x64 and x32 PC platform.
-
@derek-newbold said in Ubuntu CPU Usage High After FOG Update:
Do you have any instructions/guidance around rolling back the FOS kernel? I’m not familiar with this process and don’t want to break anything. My knowledge on linux/ubuntu systems is limited as I only use it to run FOG. We just tried to image a laptop and it seems to have got stuck at the GPT section so I’m guessing this is the issue you mentioned?
Use the built in kernel upgrade tool in FOG Settings -> FOG Kernel to install the down level kernels.
I have found a “www.conf” file in the following path, would this be the right one to edit?
“/etc/php/7.1/fpm/pool.d/www.conf”Yes. since you have php and php-fpm 7.1 installed.
As far as clients communicating. We have about 600 devices on the network that could potentially be communicating with the FOG server if they were all turned on at once. I have not had any issues with this in the past.
As mentioned in the other post, change your check in time to 600 seconds and you should see your mysql load drop off. The top processes should be the php-fpm php engines.
The devs switch to using a dedicated php engine in php-fpm and moved away from the built in apache php. This was done to improve the responsiveness of the web gui and to provide a solid and faster platform for the redesigned webgui in FOG 1.6 (still about a year off).
-
Use the built in kernel upgrade tool in FOG Settings -> FOG Kernel to install the down level kernels.
In the past I’ve had issues with the FOG server being able to connect to the internet through our departments firewall/proxy systems. I have managed to get it working this time tho thankfully and have downgraded the kernel!
I have also edited the php values and changed the client check in settings to 300. This has helped the CPU load on the server although it still seems a little high than what we are used too. At least it is a lot more responsive now on the web interface!
We are now having a GPT issue deploying an updated image that was captured after the upgrade (pre-upgrade images seem to deploy fine). I have started a new forum post on this though.
