• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Web GUI error after clicking List Hosts

    Scheduled Pinned Locked Moved
    FOG Problems
    2
    25
    4.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      tesparza
      last edited by

      I get this error after clicking List Hosts it hangs for a longtime then that error appears. We are nearing 2000+ hosts. Im guessing it a capacity issues. An ideas where to look?

      fog-error.png

      george1421G 1 Reply Last reply Reply Quote 0
      • george1421G
        george1421 Moderator @tesparza
        last edited by george1421

        @tesparza There is a short answer and a bit longer and messy one.

        Lets first check the apache error logs in /var/logs/apache2 directory. I’m going to suspect you will see a php-fpm timeout.

        Then look at the linux top utility and sort by Processor

        If your top task is mysql at about 20% or more and you see many (many) php-fpm task below. Lets stop here and let me know.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        T 2 Replies Last reply Reply Quote 0
        • T
          tesparza @george1421
          last edited by

          @george1421 mysql is running at 70.5 CPU Usage.

          [09-Nov-2021 17:28:55] WARNING: [pool www] server reached pm.max_children setting (50), consider raising it
          

          php-fpm error.log file

          1 Reply Last reply Reply Quote 0
          • T
            tesparza @george1421
            last edited by

            @george1421
            In /etc/php-fpm.d/www.conf:

            pm.max_requests = 500

            This worked.

            george1421G 1 Reply Last reply Reply Quote 0
            • george1421G
              george1421 Moderator @tesparza
              last edited by

              @tesparza said in Web GUI error after clicking List Hosts:

              pm.max_requests = 500

              You can do this but it doesn’t fix the root of the problem. What we need to do is modify your database configuration. What’s happening is the database can’t keep up with the requests so the php-fpm requests are being queued up. The workers are being depleted and eventually apache will start timing out. Setting the max requests to 500 only forces the children workers to restart after 500 requests. Restarting the workers just reboots them and discards anything they were working on like request from your FOG clients.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

              T 1 Reply Last reply Reply Quote 0
              • T
                tesparza @george1421
                last edited by

                @george1421 Okay how can I fix my database issue?

                george1421G 1 Reply Last reply Reply Quote 0
                • george1421G
                  george1421 Moderator @tesparza
                  last edited by

                  @tesparza I really need to write up a step by step on addressing this. I was working (recently) with a forum member on doing this and he ghosted me so I stopped.

                  The first we need to do is adjust a few mysql/mariadb database parameters because a few of the FOG default values will cause the migration to fail if these new settings exist with the innodb engine.

                  So first log into the mysql client program as root. Then run this command and post the results. Please post the text not a picture of the results because we are going to remove a few parameters.
                  SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session;

                  The output should look like this:

                  mysql> SELECT @@GLOBAL.sql_mode global, @@SESSION.sql_mode session;
                  +-------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
                  | global                                                                                                                                    | session                                                                                                                                   |
                  +-------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
                  | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
                  +-------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
                  1 row in set (0.01 sec)
                  

                  Once we have that we’ll go into the next step.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                  T 2 Replies Last reply Reply Quote 0
                  • T
                    tesparza @george1421
                    last edited by

                    @george1421 The command is not returning anything. Its blank Global and Session boxes

                    mysql.png

                    1 Reply Last reply Reply Quote 0
                    • T
                      tesparza @george1421
                      last edited by

                      @george1421 I copy pasted the command but its not giving the results. Im running CentOS and MariaDB

                      george1421G 1 Reply Last reply Reply Quote 0
                      • george1421G
                        george1421 Moderator @tesparza
                        last edited by

                        @tesparza Ok then you have an older version of mariadb so we can just keep moving forward with the migration.

                        I’m referencing this post: https://forums.fogproject.org/post/133490

                        At this point we need to temporarily stop apache so we don’t have any transactions going on when you run the migration command. Do this with: systemctl stop httpd on centos.

                        The next steps are to run this command via the mysql cli program:

                        SELECT TABLE_NAME,ENGINE
                        FROM information_schema.TABLES
                        WHERE TABLE_SCHEMA = 'fog' and ENGINE = 'MyISAM';
                        

                        The output should show that all tables are in MyISAM format.

                        Now paste this section into the cli. You might want to do it a few lines at a time in case there is an error message.

                        USE FOG
                        ALTER TABLE LDAPServers ENGINE=InnoDB;
                        ALTER TABLE clientUpdates ENGINE=InnoDB;
                        ALTER TABLE dirCleaner ENGINE=InnoDB;
                        ALTER TABLE globalSettings ENGINE=InnoDB;
                        ALTER TABLE greenFog ENGINE=InnoDB;
                        ALTER TABLE groupMembers ENGINE=InnoDB;
                        ALTER TABLE groups ENGINE=InnoDB;
                        ALTER TABLE history ENGINE=InnoDB;
                        ALTER TABLE hookEvents ENGINE=InnoDB;
                        ALTER TABLE hostAutoLogOut ENGINE=InnoDB;
                        ALTER TABLE hostMAC ENGINE=InnoDB;
                        ALTER TABLE hostScreenSettings ENGINE=InnoDB;
                        ALTER TABLE hosts ENGINE=InnoDB;
                        ALTER TABLE imageGroupAssoc ENGINE=InnoDB;
                        ALTER TABLE imagePartitionTypes ENGINE=InnoDB;
                        ALTER TABLE imageTypes ENGINE=InnoDB;
                        ALTER TABLE images ENGINE=InnoDB;
                        ALTER TABLE imagingLog ENGINE=InnoDB;
                        ALTER TABLE inventory ENGINE=InnoDB;
                        ALTER TABLE ipxeTable ENGINE=InnoDB;
                        ALTER TABLE keySequence ENGINE=InnoDB;
                        ALTER TABLE location ENGINE=InnoDB;
                        ALTER TABLE locationAssoc ENGINE=InnoDB;
                        ALTER TABLE moduleStatusByHost ENGINE=InnoDB;
                        ALTER TABLE modules ENGINE=InnoDB;
                        ALTER TABLE multicastSessions ENGINE=InnoDB;
                        ALTER TABLE multicastSessionsAssoc ENGINE=InnoDB;
                        ALTER TABLE nfsFailures ENGINE=InnoDB;
                        ALTER TABLE nfsGroupMembers ENGINE=InnoDB;
                        ALTER TABLE nfsGroups ENGINE=InnoDB;
                        ALTER TABLE notifyEvents ENGINE=InnoDB;
                        ALTER TABLE os ENGINE=InnoDB;
                        ALTER TABLE oui ENGINE=InnoDB;
                        ALTER TABLE plugins ENGINE=InnoDB;
                        ALTER TABLE powerManagement ENGINE=InnoDB;
                        ALTER TABLE printerAssoc ENGINE=InnoDB;
                        ALTER TABLE printers ENGINE=InnoDB;
                        ALTER TABLE pxeMenu ENGINE=InnoDB;
                        ALTER TABLE scheduledTasks ENGINE=InnoDB;
                        ALTER TABLE schemaVersion ENGINE=InnoDB;
                        ALTER TABLE snapinAssoc ENGINE=InnoDB;
                        ALTER TABLE snapinGroupAssoc ENGINE=InnoDB;
                        ALTER TABLE snapinJobs ENGINE=InnoDB;
                        ALTER TABLE snapinTasks ENGINE=InnoDB;
                        ALTER TABLE snapins ENGINE=InnoDB;
                        ALTER TABLE supportedOS ENGINE=InnoDB;
                        ALTER TABLE taskLog ENGINE=InnoDB;
                        ALTER TABLE taskStates ENGINE=InnoDB;
                        ALTER TABLE taskTypes ENGINE=InnoDB;
                        ALTER TABLE tasks ENGINE=InnoDB;
                        ALTER TABLE userCleanup ENGINE=InnoDB;
                        ALTER TABLE userTracking ENGINE=InnoDB;
                        ALTER TABLE users ENGINE=InnoDB;
                        ALTER TABLE virus ENGINE=InnoDB;
                        ALTER TABLE wolbroadcast ENGINE=InnoDB;
                        

                        And the end rerun this command

                        SELECT TABLE_NAME,ENGINE
                        FROM information_schema.TABLES
                        WHERE TABLE_SCHEMA = 'fog';
                        

                        You should now see all of the tables in innodb format. If you get an error message around an invalid date format then we will need to dig a bit deeper. I think on the debian based systems there was 3 possible tables. When I first did this on centos it worked correctly.

                        Now that all of the tables have been updated, exit the mysql cli and restart apache. systemctl start httpd

                        That should be it. Monitor top for about 10-20 minutes. You should not see 70% cpu on mysql again. Monitor for the next few hours and everything stay normal.

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                        T 1 Reply Last reply Reply Quote 0
                        • T
                          tesparza @george1421
                          last edited by

                          @george1421 Okay I did the commands, some did error out saying they dont exist but this are the results of the final command. Website seems to be running smoother. Will continue to monitor.

                          +------------------------+--------+
                          | TABLE_NAME             | ENGINE |
                          +------------------------+--------+
                          | clientUpdates          | InnoDB |
                          | dirCleaner             | InnoDB |
                          | fileChecksums          | MyISAM |
                          | globalSettings         | InnoDB |
                          | greenFog               | InnoDB |
                          | groupMembers           | InnoDB |
                          | groups                 | InnoDB |
                          | history                | InnoDB |
                          | hookEvents             | InnoDB |
                          | hostAutoLogOut         | InnoDB |
                          | hostMAC                | InnoDB |
                          | hostScreenSettings     | InnoDB |
                          | hosts                  | InnoDB |
                          | hoststatus             | MyISAM |
                          | imageGroupAssoc        | InnoDB |
                          | imagePartitionTypes    | InnoDB |
                          | imageTypes             | InnoDB |
                          | images                 | InnoDB |
                          | imagingLog             | InnoDB |
                          | inventory              | InnoDB |
                          | ipxeTable              | InnoDB |
                          | keySequence            | InnoDB |
                          | location               | InnoDB |
                          | locationAssoc          | InnoDB |
                          | moduleStatusByHost     | InnoDB |
                          | modules                | InnoDB |
                          | multicastSessions      | InnoDB |
                          | multicastSessionsAssoc | InnoDB |
                          | nfsFailures            | InnoDB |
                          | nfsGroupMembers        | InnoDB |
                          | nfsGroups              | InnoDB |
                          | notifyEvents           | InnoDB |
                          | os                     | InnoDB |
                          | oui                    | InnoDB |
                          | plugins                | InnoDB |
                          | powerManagement        | InnoDB |
                          | printerAssoc           | InnoDB |
                          | printers               | InnoDB |
                          | pxeMenu                | InnoDB |
                          | scheduledTasks         | InnoDB |
                          | schemaVersion          | InnoDB |
                          | snapinAssoc            | InnoDB |
                          | snapinGroupAssoc       | InnoDB |
                          | snapinJobs             | InnoDB |
                          | snapinTasks            | InnoDB |
                          | snapins                | InnoDB |
                          | supportedOS            | InnoDB |
                          | taskLog                | InnoDB |
                          | taskStates             | InnoDB |
                          | taskTypes              | InnoDB |
                          | tasks                  | InnoDB |
                          | userCleanup            | InnoDB |
                          | userTracking           | InnoDB |
                          | users                  | InnoDB |
                          | virus                  | InnoDB |
                          +------------------------+--------+
                          55 rows in set (0.00 sec)
                          
                          
                          
                          george1421G 1 Reply Last reply Reply Quote 0
                          • george1421G
                            george1421 Moderator @tesparza
                            last edited by

                            @tesparza OK I see two tables either I missed or there was an error converting.

                            Can you run these commands

                            use fog
                            ALTER TABLE fileChecksums ENGINE=InnoDB;
                            ALTER TABLE hoststatus ENGINE=InnoDB;
                            

                            You can do this now or later. You got the big tables that cause resource contention already, these are just for completeness.

                            I’m glad the innodb switch has settled down your FOG server, lets see how it runs for the next day or so to see if we need to tune it a bit more.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                            T 1 Reply Last reply Reply Quote 0
                            • T
                              tesparza @george1421
                              last edited by tesparza

                              @george1421 Okay now everything is InnoDB, Server is running bad, web gui taking long time to load, can’t image. CPU usage is around 40%

                              FYI this is running of a Optiplex 7010 i5 quad core, 8gb ram

                              Do I maybe need to switch to a server?

                              george1421G 1 Reply Last reply Reply Quote 0
                              • george1421G
                                george1421 Moderator @tesparza
                                last edited by george1421

                                @tesparza Ok the innodb engine is much more efficient than myisam, so the effort is not wasted. It will never be worse with innodb format.

                                So what is your top process now? Post a screen shot of top sorted by Process.

                                How many computers with FOG clients?
                                What is your client check in interval? Still the default of 5 seconds?

                                Don’t change anything lets see where we are at first.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                T 1 Reply Last reply Reply Quote 0
                                • T
                                  tesparza @george1421
                                  last edited by

                                  @george1421 process.PNG

                                  2000 + clients

                                  Interval checkin time it 60 seconds

                                  george1421G 1 Reply Last reply Reply Quote 0
                                  • george1421G
                                    george1421 Moderator @tesparza
                                    last edited by george1421

                                    @tesparza For testing lets change the check in interval to 300 seconds (5 minutes) and then lets wait 15 minutes to see if things settle down. This isn’t the answer at the moment, its just to see if having 2000 hosts hit (if we lineralize it, 34 per second) is a bit too much for your hardware.

                                    The screen shot you shown is indicative of what a myisam engine configuration will give us. We know we already fixed that.

                                    Looking at top, it also appears you only have 4GB of ram. Can you do a uname -a and post the results here?

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                    T 1 Reply Last reply Reply Quote 0
                                    • T
                                      tesparza @george1421
                                      last edited by

                                      @george1421 Okay interval changed to 300

                                      uname.PNG

                                      george1421G 1 Reply Last reply Reply Quote 0
                                      • george1421G
                                        george1421 Moderator @tesparza
                                        last edited by george1421

                                        @tesparza well that’s a good thing you have a 64 bit OS. Since you are telling me 8GB is installed but top is reporting only 4GB then I was thinking a 32 bit OS.

                                        If the 300 second check in doesn’t settle things down, then please confirm you have 8GB installed. Since is a 7010 you might be able to find some old memory to top it out at 16GB. I know I have at least 10 sticks of 4GB 7010 memory in the bottom of my junk bin. I’m hoping you have similar.

                                        Again when things settle down a bit we’ll check also disk performance. Right now the sql sever is having a high cpu load. This could be either disk, or not enough free ram for caching to happen. We need to make small changes to see where things pick up the most.

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                        T 1 Reply Last reply Reply Quote 0
                                        • T
                                          tesparza @george1421
                                          last edited by

                                          @george1421 Okay it seems to be running smoother, I will add more RAM, eventually I will transfer to a server.

                                          george1421G 1 Reply Last reply Reply Quote 0
                                          • george1421G
                                            george1421 Moderator @tesparza
                                            last edited by

                                            @tesparza I made a post a few years ago to test different hardware and its impact on imaging speed. We may need to reference it as we work through this.

                                            https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast

                                            If your fog server has a sata spinning hard drive that may also be the culprit. But again testing will show us.

                                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                            T 1 Reply Last reply Reply Quote 1
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post

                                            195

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project