• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    High CPU Usage on Fog Server

    Scheduled Pinned Locked Moved
    FOG Problems
    3
    19
    1.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • george1421G
      george1421 Moderator @DJslimD1k
      last edited by george1421

      @DJslimD1k ok so the bottleneck is in mysql. We were researching this a bit ago and decided to move from ISAM format to innodb format. That changed the locking structure on that database.

      I also see quite a few php-fpm workers stacked up waiting for the database (guess).

      Also you have 22 vCPUs on this fog server??? How many physical cores you do have in this server? Off the top, I would say that many vCPUs is actually killing performance and not helping you one bit unless you have 96 physical cores in that vm host server.

      the innodb part is discussed here: https://forums.fogproject.org/topic/14254/high-cpu-usage/24

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

      DJslimD1kD 1 Reply Last reply Reply Quote 0
      • DJslimD1kD
        DJslimD1k @george1421
        last edited by

        @george1421 When I followed the innodb steps on the link you sent me, I got some errors. For example, on the ALTER TABLE for tasks, I get this error “ERROR 1292 (22007): Incorrect datetime value: ‘0000-00-00 00:00:00’ for column ‘ilFinishTime’ at row 2599” These are all of the effected ones that did not change. Any suggestions on how to fix this? CPU usage still seems to be high.

        54292a1d-19f9-4513-b80e-59f0ceb884a4-image.png

        george1421G 1 Reply Last reply Reply Quote 0
        • george1421G
          george1421 Moderator @DJslimD1k
          last edited by george1421

          @DJslimD1k Sorry I got pulled in another direction. On the updated versions of mariadb they made a default setting that isn’t compatible with FOG’s default one sec

          Edit: You need update a mysql configuration file here: https://forums.fogproject.org/post/134207

          Depending on your install version it might be in another file but in the section [mysqld] you need to add the sql-mode line then restart the database server. Once restarted then you should be able to convert the rest of the tables.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

          DJslimD1kD 1 Reply Last reply Reply Quote 0
          • DJslimD1kD
            DJslimD1k @george1421
            last edited by

            @george1421 So I inserted the file that you sent the link to in the /var/lib/mysql folder. I tried and followed all of the steps and I’m still getting this error when I the script to change the type.
            I’m using CentOS8 and running Fog server version 1.5.9-RC2.11

            99a27876-e464-4083-8ed1-4e5e34255611-image.png
            e239f464-ecd3-477e-a184-27c550b5e3d6-image.png
            a3b1d07a-aa38-44cc-b170-effce8cbebf5-image.png
            975251fc-4428-4ec5-856d-033f81175bc9-image.png

            george1421G 1 Reply Last reply Reply Quote 0
            • S
              Sebastian Roth Moderator
              last edited by

              @DJslimD1k After setting the config did you restart the MySQL/MariaDB?

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              1 Reply Last reply Reply Quote 0
              • george1421G
                george1421 Moderator @DJslimD1k
                last edited by

                @DJslimD1k Run the following command on the mysql server console and post the results SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session;

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                DJslimD1kD 1 Reply Last reply Reply Quote 0
                • DJslimD1kD
                  DJslimD1k @george1421
                  last edited by

                  @george1421 Still no luck. Here is the screenshot you requested.

                  794ff348-9770-4820-b29c-1ec3f0e0c2c3-image.png

                  george1421G 3 Replies Last reply Reply Quote 0
                  • george1421G
                    george1421 Moderator @DJslimD1k
                    last edited by

                    @DJslimD1k Ok this specifically is the problem.

                    date_error.png

                    So your current FOG server OS is centos 8? We need to find which file the stanza of [mysqld] is located and add in

                    sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
                    

                    and then restart mysql/mariadb. When we get it right those parameters should be gone from the query.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                    1 Reply Last reply Reply Quote 0
                    • george1421G
                      george1421 Moderator @DJslimD1k
                      last edited by george1421

                      @DJslimD1k First/accepted answer here: https://stackoverflow.com/questions/16603947/mysql-mode-on-linux-centos As to the location where that parameter needs to be set.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                      1 Reply Last reply Reply Quote 0
                      • george1421G
                        george1421 Moderator @DJslimD1k
                        last edited by george1421

                        @DJslimD1k I have a Centos 8 fog server in my home lab and this is what I had to do.

                        Edit this file
                        /etc/my.cnf.d/mysql-server.cnf

                        Add this in just below the [mysqld] section header. NOTE: This is a different sql-mode values than I gave you before. Those did not seem to work on the mysqld that centos uses. I queried the values that mysqld used and then just removed the date invalid and zero switches.

                        sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"

                        The config file should look like this:

                        [mysqld]
                        sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"
                        
                        datadir=/var/lib/mysql
                        socket=/var/lib/mysql/mysql.sock
                        log-error=/var/log/mysql/mysqld.log
                        pid-file=/run/mysqld/mysqld.pid
                        
                        

                        Restart mysqld
                        systemctl restart mysqld

                        Then query using mysql untility to ensure that the date settings are removed as in the picture from the previous post. Once that is done you should be able to convert the remaining tables to innodb format.

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                        DJslimD1kD 1 Reply Last reply Reply Quote 0
                        • DJslimD1kD
                          DJslimD1k @george1421
                          last edited by

                          @george1421 I had to change the permissions of that file in order for me to add the text into that file. Now when I log into fog, I get this message.
                          1beb019c-fcf4-419c-9caa-eaf8c73acf86-image.png

                          This is the result.

                          f06b0e0c-67f2-4841-8896-6e8c810522fc-image.png

                          george1421G 1 Reply Last reply Reply Quote 0
                          • george1421G
                            george1421 Moderator @DJslimD1k
                            last edited by

                            @DJslimD1k Well its because you changed the permissions on that file. You should have edited it with root user or sudo edited the file. Change the mode back to 755 sudo chmod 755 /etc/my.cnf.d/mysql-server.cnf Then restart mysql server.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                            DJslimD1kD 1 Reply Last reply Reply Quote 1
                            • DJslimD1kD
                              DJslimD1k @george1421
                              last edited by

                              @george1421 I believe I got it! The overall processor usage has dropped, but I still see a lot of php-fpm in the “top” view in terminal. Thank you so much for your help!

                              f88c870d-a351-4db6-b742-fb756902e1f3-image.png

                              I do have the access control plugin installed on our fog server. I figured that’s what the remaining MyISAM are from.

                              eb0811a1-5522-4ac0-9fd6-ecd47137903c-image.png

                              george1421G 1 Reply Last reply Reply Quote 0
                              • george1421G
                                george1421 Moderator @DJslimD1k
                                last edited by

                                @DJslimD1k the mysql value is still pretty high. Higher than I might expect. But switching from isam to innodb helps because isam uses table locking on update and innodb uses row locking. With a lot of clients polling that table locking method becomes a road block.

                                I didn’t look back in the tread, but didn’t your fog server have 10 vCPUs? If so I would drop those back to 4 and see where performance settles out to. Overcommiting on vCPU is worse then not providing enough vCPUs to handle the load.

                                Having that many php-fpm workers just means that your server is busy with requests. There is some performance tuning you can do with the php-fpm settings on a busy network by switching from dynamic to allocation to static allocation.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                1 Reply Last reply Reply Quote 0
                                • 1 / 1
                                • First post
                                  Last post

                                135

                                Online

                                12.1k

                                Users

                                17.3k

                                Topics

                                155.3k

                                Posts
                                Copyright © 2012-2024 FOG Project