UNSOLVED FOG Server High CPU
@fry_p Did you restart the fog server after making those adjustments?
The thing I noticed is that mysql usage is higher than php-fpm. What I’m thinking is that mysql is slow for some reason, causing php-fpm to detect a slow responding child, so it spins up more children until 35 are running.
What was your client check in time? With 1300 clients, I might think about changing the check in time to 300 (5 min) or 600 (10 min). At 300 seconds (if you average the check ins out) that 4 devices registering every second. We all know that check ins are random based on the population. So you may have 15-20 check in at once, in one second and nothing for 3 seconds after.
Most of these settings were set properly. Below are the ones I set:
pm.max_children = 35 php_admin_value[memory_limit] = 256M
A reboot after that still yields 100% usage and way too many php-fpm processes.
I have about 1300 devices using the FOG client. I realize this is a lot, so I did set the checkin time to 180s. Do you think increasing that more would help?
@fry_p Well the first thing I noticed is that you have way too many php-fpm processes running. Typically we see 5-7 process running at one time.
Below is my standard response when it comes to FOG between version 1.5.2 and 1.5.4. Ensure your settings are consistent with these:
Lets assume is the issue we’ve found after FOG 1.5.4 has been released.
- Change to the /etc directory from the fog server linux command prompt.
- Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
find /etc -name www.conf(hopefully you will only find one)
- Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except
php_admin_value[memory_limit] = 256Myou will need to add that entry.
php_admin_value[memory_limit] = 256M pm.max_requests = 2000 pm.max_children = 35 pm.min_spare_servers = 5 pm.start_servers = 5
- Save and exit your text editor.
- Reboot the fog server.
- See if that fixes what is wrong. You really should only see this strangeness under heavy load, but I guess it might show up sooner under certain conditions.
Also we found there is something strange going on in the linux kernels after 4.15.2, I’m going to recommend that you downgrade your FOG/FOS kernel to 4.15.2. The issue with later kernels is that its taking 3-5 minutes to create the disk structure under certain circumstances, where with 4.15.2 and older its only seconds to create the structure.
Now the kernel will not impact your issue, but processing is incomplete might be related to the missing php-fpm configuration setting.
For completeness, how many computers that have the fog client installed, connect to this fog server. (round numbers are good).