• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

High CPU Usage on Fog Server

Scheduled Pinned Locked Moved
FOG Problems
3
19
1.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • G
    george1421 Moderator @DJslimD1k
    last edited by george1421 Aug 25, 2020, 9:36 AM Aug 25, 2020, 3:33 PM

    @DJslimD1k ok so the bottleneck is in mysql. We were researching this a bit ago and decided to move from ISAM format to innodb format. That changed the locking structure on that database.

    I also see quite a few php-fpm workers stacked up waiting for the database (guess).

    Also you have 22 vCPUs on this fog server??? How many physical cores you do have in this server? Off the top, I would say that many vCPUs is actually killing performance and not helping you one bit unless you have 96 physical cores in that vm host server.

    the innodb part is discussed here: https://forums.fogproject.org/topic/14254/high-cpu-usage/24

    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

    D 1 Reply Last reply Aug 25, 2020, 5:54 PM Reply Quote 0
    • D
      DJslimD1k @george1421
      last edited by Aug 25, 2020, 5:54 PM

      @george1421 When I followed the innodb steps on the link you sent me, I got some errors. For example, on the ALTER TABLE for tasks, I get this error “ERROR 1292 (22007): Incorrect datetime value: ‘0000-00-00 00:00:00’ for column ‘ilFinishTime’ at row 2599” These are all of the effected ones that did not change. Any suggestions on how to fix this? CPU usage still seems to be high.

      54292a1d-19f9-4513-b80e-59f0ceb884a4-image.png

      G 1 Reply Last reply Aug 25, 2020, 5:59 PM Reply Quote 0
      • G
        george1421 Moderator @DJslimD1k
        last edited by george1421 Aug 25, 2020, 12:01 PM Aug 25, 2020, 5:59 PM

        @DJslimD1k Sorry I got pulled in another direction. On the updated versions of mariadb they made a default setting that isn’t compatible with FOG’s default one sec

        Edit: You need update a mysql configuration file here: https://forums.fogproject.org/post/134207

        Depending on your install version it might be in another file but in the section [mysqld] you need to add the sql-mode line then restart the database server. Once restarted then you should be able to convert the rest of the tables.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        D 1 Reply Last reply Aug 25, 2020, 8:56 PM Reply Quote 0
        • D
          DJslimD1k @george1421
          last edited by Aug 25, 2020, 8:56 PM

          @george1421 So I inserted the file that you sent the link to in the /var/lib/mysql folder. I tried and followed all of the steps and I’m still getting this error when I the script to change the type.
          I’m using CentOS8 and running Fog server version 1.5.9-RC2.11

          99a27876-e464-4083-8ed1-4e5e34255611-image.png
          e239f464-ecd3-477e-a184-27c550b5e3d6-image.png
          a3b1d07a-aa38-44cc-b170-effce8cbebf5-image.png
          975251fc-4428-4ec5-856d-033f81175bc9-image.png

          G 1 Reply Last reply Aug 26, 2020, 1:14 PM Reply Quote 0
          • S
            Sebastian Roth Moderator
            last edited by Aug 26, 2020, 9:38 AM

            @DJslimD1k After setting the config did you restart the MySQL/MariaDB?

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            1 Reply Last reply Reply Quote 0
            • G
              george1421 Moderator @DJslimD1k
              last edited by Aug 26, 2020, 1:14 PM

              @DJslimD1k Run the following command on the mysql server console and post the results SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session;

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

              D 1 Reply Last reply Aug 28, 2020, 1:37 PM Reply Quote 0
              • D
                DJslimD1k @george1421
                last edited by Aug 28, 2020, 1:37 PM

                @george1421 Still no luck. Here is the screenshot you requested.

                794ff348-9770-4820-b29c-1ec3f0e0c2c3-image.png

                G 3 Replies Last reply Aug 28, 2020, 1:40 PM Reply Quote 0
                • G
                  george1421 Moderator @DJslimD1k
                  last edited by Aug 28, 2020, 1:40 PM

                  @DJslimD1k Ok this specifically is the problem.

                  date_error.png

                  So your current FOG server OS is centos 8? We need to find which file the stanza of [mysqld] is located and add in

                  sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
                  

                  and then restart mysql/mariadb. When we get it right those parameters should be gone from the query.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                  1 Reply Last reply Reply Quote 0
                  • G
                    george1421 Moderator @DJslimD1k
                    last edited by george1421 Aug 28, 2020, 7:44 AM Aug 28, 2020, 1:43 PM

                    @DJslimD1k First/accepted answer here: https://stackoverflow.com/questions/16603947/mysql-mode-on-linux-centos As to the location where that parameter needs to be set.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                    1 Reply Last reply Reply Quote 0
                    • G
                      george1421 Moderator @DJslimD1k
                      last edited by george1421 Aug 29, 2020, 7:50 AM Aug 29, 2020, 1:48 PM

                      @DJslimD1k I have a Centos 8 fog server in my home lab and this is what I had to do.

                      Edit this file
                      /etc/my.cnf.d/mysql-server.cnf

                      Add this in just below the [mysqld] section header. NOTE: This is a different sql-mode values than I gave you before. Those did not seem to work on the mysqld that centos uses. I queried the values that mysqld used and then just removed the date invalid and zero switches.

                      sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"

                      The config file should look like this:

                      [mysqld]
                      sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"
                      
                      datadir=/var/lib/mysql
                      socket=/var/lib/mysql/mysql.sock
                      log-error=/var/log/mysql/mysqld.log
                      pid-file=/run/mysqld/mysqld.pid
                      
                      

                      Restart mysqld
                      systemctl restart mysqld

                      Then query using mysql untility to ensure that the date settings are removed as in the picture from the previous post. Once that is done you should be able to convert the remaining tables to innodb format.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                      D 1 Reply Last reply Sep 2, 2020, 11:28 PM Reply Quote 0
                      • D
                        DJslimD1k @george1421
                        last edited by Sep 2, 2020, 11:28 PM

                        @george1421 I had to change the permissions of that file in order for me to add the text into that file. Now when I log into fog, I get this message.
                        1beb019c-fcf4-419c-9caa-eaf8c73acf86-image.png

                        This is the result.

                        f06b0e0c-67f2-4841-8896-6e8c810522fc-image.png

                        G 1 Reply Last reply Sep 3, 2020, 12:00 AM Reply Quote 0
                        • G
                          george1421 Moderator @DJslimD1k
                          last edited by Sep 3, 2020, 12:00 AM

                          @DJslimD1k Well its because you changed the permissions on that file. You should have edited it with root user or sudo edited the file. Change the mode back to 755 sudo chmod 755 /etc/my.cnf.d/mysql-server.cnf Then restart mysql server.

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                          D 1 Reply Last reply Sep 3, 2020, 1:33 PM Reply Quote 1
                          • D
                            DJslimD1k @george1421
                            last edited by Sep 3, 2020, 1:33 PM

                            @george1421 I believe I got it! The overall processor usage has dropped, but I still see a lot of php-fpm in the “top” view in terminal. Thank you so much for your help!

                            f88c870d-a351-4db6-b742-fb756902e1f3-image.png

                            I do have the access control plugin installed on our fog server. I figured that’s what the remaining MyISAM are from.

                            eb0811a1-5522-4ac0-9fd6-ecd47137903c-image.png

                            G 1 Reply Last reply Sep 3, 2020, 2:30 PM Reply Quote 0
                            • G
                              george1421 Moderator @DJslimD1k
                              last edited by Sep 3, 2020, 2:30 PM

                              @DJslimD1k the mysql value is still pretty high. Higher than I might expect. But switching from isam to innodb helps because isam uses table locking on update and innodb uses row locking. With a lot of clients polling that table locking method becomes a road block.

                              I didn’t look back in the tread, but didn’t your fog server have 10 vCPUs? If so I would drop those back to 4 and see where performance settles out to. Overcommiting on vCPU is worse then not providing enough vCPUs to handle the load.

                              Having that many php-fpm workers just means that your server is busy with requests. There is some performance tuning you can do with the php-fpm settings on a busy network by switching from dynamic to allocation to static allocation.

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                              1 Reply Last reply Reply Quote 0
                              • 1 / 1
                              1 / 1
                              • First post
                                15/19
                                Last post

                              145

                              Online

                              12.0k

                              Users

                              17.3k

                              Topics

                              155.2k

                              Posts
                              Copyright © 2012-2024 FOG Project