High CPU Usage on Fog Server

[DJslimD1k]

Hello everyone! Now that school is back in session and all of the computers in the school district are turned on, we noticed the CPU usage on the server has been extremely high. The only thing that we could think that would cause this is all of those machines are now constantly checking into the server. Is there a way to have the clients check less often? The only thing we use the fog client for is to join the hosts to the domain. Any suggestions?

[george1421]

FOG Configuration -> FOG Settings -> FOG Client -> CLIENT CHECKIN TIME

Change time to 900 (15 minutes).

[DJslimD1k]

Sweet! Thank you very much! If the processor usage remains high, do you have anymore recommendations? I currently have 6 storage nodes in place to help with the deployment process. The main server seems to be struggling the most.

[george1421]

@DJslimD1k Well this may need a bit more investigation.

On your fog server when you run the top program and sort by P processor/cpu what is the top process? Possibly a screen shot of the top page would be useful.

How many computers have the FOG Client installed on them?

[DJslimD1k]

@george1421 Currently, mysql is using 1,415% of the processor. That doesn’t seem to be good. We have about 2,000 machines with the fog client installed. The main fog server has 32 GB of ram with 22 logical processors on it. My 6 storage nodes have 16gb of ram with 4 logical processors.

[george1421]

@DJslimD1k ok so the bottleneck is in mysql. We were researching this a bit ago and decided to move from ISAM format to innodb format. That changed the locking structure on that database.

I also see quite a few php-fpm workers stacked up waiting for the database (guess).

Also you have 22 vCPUs on this fog server??? How many physical cores you do have in this server? Off the top, I would say that many vCPUs is actually killing performance and not helping you one bit unless you have 96 physical cores in that vm host server.

the innodb part is discussed here: https://forums.fogproject.org/topic/14254/high-cpu-usage/24

[DJslimD1k]

@george1421 When I followed the innodb steps on the link you sent me, I got some errors. For example, on the ALTER TABLE for tasks, I get this error “ERROR 1292 (22007): Incorrect datetime value: ‘0000-00-00 00:00:00’ for column ‘ilFinishTime’ at row 2599” These are all of the effected ones that did not change. Any suggestions on how to fix this? CPU usage still seems to be high.

[george1421]

@DJslimD1k Sorry I got pulled in another direction. On the updated versions of mariadb they made a default setting that isn’t compatible with FOG’s default one sec

Edit: You need update a mysql configuration file here: https://forums.fogproject.org/post/134207

Depending on your install version it might be in another file but in the section [mysqld] you need to add the sql-mode line then restart the database server. Once restarted then you should be able to convert the rest of the tables.

[DJslimD1k]

@george1421 So I inserted the file that you sent the link to in the /var/lib/mysql folder. I tried and followed all of the steps and I’m still getting this error when I the script to change the type.
I’m using CentOS8 and running Fog server version 1.5.9-RC2.11

[Sebastian Roth]

@DJslimD1k After setting the config did you restart the MySQL/MariaDB?

[george1421]

@DJslimD1k Run the following command on the mysql server console and post the results SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session;

[DJslimD1k]

@george1421 Still no luck. Here is the screenshot you requested.

[george1421]

@DJslimD1k Ok this specifically is the problem.

So your current FOG server OS is centos 8? We need to find which file the stanza of [mysqld] is located and add in

sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

and then restart mysql/mariadb. When we get it right those parameters should be gone from the query.

[george1421]

@DJslimD1k First/accepted answer here: https://stackoverflow.com/questions/16603947/mysql-mode-on-linux-centos As to the location where that parameter needs to be set.

[george1421]

@DJslimD1k I have a Centos 8 fog server in my home lab and this is what I had to do.

Edit this file
/etc/my.cnf.d/mysql-server.cnf

Add this in just below the [mysqld] section header. NOTE: This is a different sql-mode values than I gave you before. Those did not seem to work on the mysqld that centos uses. I queried the values that mysqld used and then just removed the date invalid and zero switches.

sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"

The config file should look like this:

[mysqld]
sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysql/mysqld.log
pid-file=/run/mysqld/mysqld.pid

Restart mysqld
systemctl restart mysqld

Then query using mysql untility to ensure that the date settings are removed as in the picture from the previous post. Once that is done you should be able to convert the remaining tables to innodb format.

[DJslimD1k]

@george1421 I had to change the permissions of that file in order for me to add the text into that file. Now when I log into fog, I get this message.

This is the result.

[george1421]

@DJslimD1k Well its because you changed the permissions on that file. You should have edited it with root user or sudo edited the file. Change the mode back to 755 sudo chmod 755 /etc/my.cnf.d/mysql-server.cnf Then restart mysql server.

[DJslimD1k]

@george1421 I believe I got it! The overall processor usage has dropped, but I still see a lot of php-fpm in the “top” view in terminal. Thank you so much for your help!

I do have the access control plugin installed on our fog server. I figured that’s what the remaining MyISAM are from.

[george1421]

@DJslimD1k the mysql value is still pretty high. Higher than I might expect. But switching from isam to innodb helps because isam uses table locking on update and innodb uses row locking. With a lot of clients polling that table locking method becomes a road block.

I didn’t look back in the tread, but didn’t your fog server have 10 vCPUs? If so I would drop those back to 4 and see where performance settles out to. Overcommiting on vCPU is worse then not providing enough vCPUs to handle the load.

Having that many php-fpm workers just means that your server is busy with requests. There is some performance tuning you can do with the php-fpm settings on a busy network by switching from dynamic to allocation to static allocation.