• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    High CPU Usage on Fog Server

    Scheduled Pinned Locked Moved
    FOG Problems
    3
    19
    1.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DJslimD1kD
      DJslimD1k
      last edited by

      Hello everyone! Now that school is back in session and all of the computers in the school district are turned on, we noticed the CPU usage on the server has been extremely high. The only thing that we could think that would cause this is all of those machines are now constantly checking into the server. Is there a way to have the clients check less often? The only thing we use the fog client for is to join the hosts to the domain. Any suggestions?

      1 Reply Last reply Reply Quote 0
      • george1421G
        george1421 Moderator
        last edited by

        FOG Configuration -> FOG Settings -> FOG Client -> CLIENT CHECKIN TIME

        Change time to 900 (15 minutes).

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        1 Reply Last reply Reply Quote 0
        • DJslimD1kD
          DJslimD1k
          last edited by DJslimD1k

          Sweet! Thank you very much! If the processor usage remains high, do you have anymore recommendations? I currently have 6 storage nodes in place to help with the deployment process. The main server seems to be struggling the most.

          george1421G 1 Reply Last reply Reply Quote 0
          • george1421G
            george1421 Moderator @DJslimD1k
            last edited by george1421

            @DJslimD1k Well this may need a bit more investigation.

            On your fog server when you run the top program and sort by P processor/cpu what is the top process? Possibly a screen shot of the top page would be useful.

            How many computers have the FOG Client installed on them?

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

            DJslimD1kD 1 Reply Last reply Reply Quote 0
            • DJslimD1kD
              DJslimD1k @george1421
              last edited by DJslimD1k

              @george1421 Currently, mysql is using 1,415% of the processor. That doesn’t seem to be good. We have about 2,000 machines with the fog client installed. The main fog server has 32 GB of ram with 22 logical processors on it. My 6 storage nodes have 16gb of ram with 4 logical processors.

              9fe7176a-de46-4709-9c1a-fd749fa9594a-image.png

              george1421G 1 Reply Last reply Reply Quote 0
              • george1421G
                george1421 Moderator @DJslimD1k
                last edited by george1421

                @DJslimD1k ok so the bottleneck is in mysql. We were researching this a bit ago and decided to move from ISAM format to innodb format. That changed the locking structure on that database.

                I also see quite a few php-fpm workers stacked up waiting for the database (guess).

                Also you have 22 vCPUs on this fog server??? How many physical cores you do have in this server? Off the top, I would say that many vCPUs is actually killing performance and not helping you one bit unless you have 96 physical cores in that vm host server.

                the innodb part is discussed here: https://forums.fogproject.org/topic/14254/high-cpu-usage/24

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                DJslimD1kD 1 Reply Last reply Reply Quote 0
                • DJslimD1kD
                  DJslimD1k @george1421
                  last edited by

                  @george1421 When I followed the innodb steps on the link you sent me, I got some errors. For example, on the ALTER TABLE for tasks, I get this error “ERROR 1292 (22007): Incorrect datetime value: ‘0000-00-00 00:00:00’ for column ‘ilFinishTime’ at row 2599” These are all of the effected ones that did not change. Any suggestions on how to fix this? CPU usage still seems to be high.

                  54292a1d-19f9-4513-b80e-59f0ceb884a4-image.png

                  george1421G 1 Reply Last reply Reply Quote 0
                  • george1421G
                    george1421 Moderator @DJslimD1k
                    last edited by george1421

                    @DJslimD1k Sorry I got pulled in another direction. On the updated versions of mariadb they made a default setting that isn’t compatible with FOG’s default one sec

                    Edit: You need update a mysql configuration file here: https://forums.fogproject.org/post/134207

                    Depending on your install version it might be in another file but in the section [mysqld] you need to add the sql-mode line then restart the database server. Once restarted then you should be able to convert the rest of the tables.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                    DJslimD1kD 1 Reply Last reply Reply Quote 0
                    • DJslimD1kD
                      DJslimD1k @george1421
                      last edited by

                      @george1421 So I inserted the file that you sent the link to in the /var/lib/mysql folder. I tried and followed all of the steps and I’m still getting this error when I the script to change the type.
                      I’m using CentOS8 and running Fog server version 1.5.9-RC2.11

                      99a27876-e464-4083-8ed1-4e5e34255611-image.png
                      e239f464-ecd3-477e-a184-27c550b5e3d6-image.png
                      a3b1d07a-aa38-44cc-b170-effce8cbebf5-image.png
                      975251fc-4428-4ec5-856d-033f81175bc9-image.png

                      george1421G 1 Reply Last reply Reply Quote 0
                      • S
                        Sebastian Roth Moderator
                        last edited by

                        @DJslimD1k After setting the config did you restart the MySQL/MariaDB?

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        1 Reply Last reply Reply Quote 0
                        • george1421G
                          george1421 Moderator @DJslimD1k
                          last edited by

                          @DJslimD1k Run the following command on the mysql server console and post the results SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session;

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                          DJslimD1kD 1 Reply Last reply Reply Quote 0
                          • DJslimD1kD
                            DJslimD1k @george1421
                            last edited by

                            @george1421 Still no luck. Here is the screenshot you requested.

                            794ff348-9770-4820-b29c-1ec3f0e0c2c3-image.png

                            george1421G 3 Replies Last reply Reply Quote 0
                            • george1421G
                              george1421 Moderator @DJslimD1k
                              last edited by

                              @DJslimD1k Ok this specifically is the problem.

                              date_error.png

                              So your current FOG server OS is centos 8? We need to find which file the stanza of [mysqld] is located and add in

                              sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
                              

                              and then restart mysql/mariadb. When we get it right those parameters should be gone from the query.

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                              1 Reply Last reply Reply Quote 0
                              • george1421G
                                george1421 Moderator @DJslimD1k
                                last edited by george1421

                                @DJslimD1k First/accepted answer here: https://stackoverflow.com/questions/16603947/mysql-mode-on-linux-centos As to the location where that parameter needs to be set.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                1 Reply Last reply Reply Quote 0
                                • george1421G
                                  george1421 Moderator @DJslimD1k
                                  last edited by george1421

                                  @DJslimD1k I have a Centos 8 fog server in my home lab and this is what I had to do.

                                  Edit this file
                                  /etc/my.cnf.d/mysql-server.cnf

                                  Add this in just below the [mysqld] section header. NOTE: This is a different sql-mode values than I gave you before. Those did not seem to work on the mysqld that centos uses. I queried the values that mysqld used and then just removed the date invalid and zero switches.

                                  sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"

                                  The config file should look like this:

                                  [mysqld]
                                  sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"
                                  
                                  datadir=/var/lib/mysql
                                  socket=/var/lib/mysql/mysql.sock
                                  log-error=/var/log/mysql/mysqld.log
                                  pid-file=/run/mysqld/mysqld.pid
                                  
                                  

                                  Restart mysqld
                                  systemctl restart mysqld

                                  Then query using mysql untility to ensure that the date settings are removed as in the picture from the previous post. Once that is done you should be able to convert the remaining tables to innodb format.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                  DJslimD1kD 1 Reply Last reply Reply Quote 0
                                  • DJslimD1kD
                                    DJslimD1k @george1421
                                    last edited by

                                    @george1421 I had to change the permissions of that file in order for me to add the text into that file. Now when I log into fog, I get this message.
                                    1beb019c-fcf4-419c-9caa-eaf8c73acf86-image.png

                                    This is the result.

                                    f06b0e0c-67f2-4841-8896-6e8c810522fc-image.png

                                    george1421G 1 Reply Last reply Reply Quote 0
                                    • george1421G
                                      george1421 Moderator @DJslimD1k
                                      last edited by

                                      @DJslimD1k Well its because you changed the permissions on that file. You should have edited it with root user or sudo edited the file. Change the mode back to 755 sudo chmod 755 /etc/my.cnf.d/mysql-server.cnf Then restart mysql server.

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                      DJslimD1kD 1 Reply Last reply Reply Quote 1
                                      • DJslimD1kD
                                        DJslimD1k @george1421
                                        last edited by

                                        @george1421 I believe I got it! The overall processor usage has dropped, but I still see a lot of php-fpm in the “top” view in terminal. Thank you so much for your help!

                                        f88c870d-a351-4db6-b742-fb756902e1f3-image.png

                                        I do have the access control plugin installed on our fog server. I figured that’s what the remaining MyISAM are from.

                                        eb0811a1-5522-4ac0-9fd6-ecd47137903c-image.png

                                        george1421G 1 Reply Last reply Reply Quote 0
                                        • george1421G
                                          george1421 Moderator @DJslimD1k
                                          last edited by

                                          @DJslimD1k the mysql value is still pretty high. Higher than I might expect. But switching from isam to innodb helps because isam uses table locking on update and innodb uses row locking. With a lot of clients polling that table locking method becomes a road block.

                                          I didn’t look back in the tread, but didn’t your fog server have 10 vCPUs? If so I would drop those back to 4 and see where performance settles out to. Overcommiting on vCPU is worse then not providing enough vCPUs to handle the load.

                                          Having that many php-fpm workers just means that your server is busy with requests. There is some performance tuning you can do with the php-fpm settings on a busy network by switching from dynamic to allocation to static allocation.

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                          1 Reply Last reply Reply Quote 0
                                          • 1 / 1
                                          • First post
                                            Last post

                                          187

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project