• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

FOG Web GUI speed and default storage activity

Scheduled Pinned Locked Moved Solved
FOG Problems
5
62
22.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J
    JGallo @Wayne Workman
    last edited by JGallo Oct 18, 2017, 8:30 AM Oct 18, 2017, 2:29 PM

    @wayne-workman

    Came in and checked FOG Web UI and messages still persists. I checked server and cache is at 571MB. I will say this, it definitely fixed the web UI speeds by clearing the cache.

    W 1 Reply Last reply Oct 19, 2017, 12:23 AM Reply Quote 0
    • W
      Wayne Workman @JGallo
      last edited by Oct 19, 2017, 12:23 AM

      @jgallo Try clearing the cache the same way on one of the nodes, see if it fixes that node.

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
      Daily Clean Installation Results:
      https://fogtesting.fogproject.us/
      FOG Reporting:
      https://fog-external-reporting-results.fogproject.us/

      J 1 Reply Last reply Oct 19, 2017, 3:43 PM Reply Quote 0
      • J
        JGallo @Wayne Workman
        last edited by Oct 19, 2017, 3:43 PM

        @wayne-workman

        I have done that to all the nodes I have the graph enabled to. Rebooted server and observing if it helps. How fast should the buff/cache be growing? I’m noticing on a few nodes that it spikes really fast and some other nodes it’s just normalized around 1GB even though they have 16GB. For example one of my nodes has 30GB of memory available and I cleared the cache. Within 1 minute it grew over 5GB and running another minute it has grown to 10GB.

        W 1 Reply Last reply Oct 19, 2017, 5:36 PM Reply Quote 0
        • W
          Wayne Workman @JGallo
          last edited by Oct 19, 2017, 5:36 PM

          @jgallo Having cache in the RAM isn’t a bad thing - Linux does a really amazing job at managing memory, it’s just something about fog that causes the cache to not be effective (what’s cached is not what’s actually used often). cache size grows as the Linux kernel does stuff. The less stuff it does, the slower the cache grows. Eventually though - and by design - the RAM gets filled with cache in order to make future requests faster. Again, in the case of fog, what’s cached is not what is getting used over and over.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
          Daily Clean Installation Results:
          https://fogtesting.fogproject.us/
          FOG Reporting:
          https://fog-external-reporting-results.fogproject.us/

          J 1 Reply Last reply Oct 19, 2017, 5:47 PM Reply Quote 0
          • J
            JGallo @Wayne Workman
            last edited by Oct 19, 2017, 5:47 PM

            @wayne-workman

            So far it has not prompted me with database connections. earlier I have cleared the cache on all nodes and rebooted. Seems like it’s working because so far it has not prompted anything with database connections errors and I have been sitting on the dashboard since.

            W 1 Reply Last reply Oct 19, 2017, 5:48 PM Reply Quote 0
            • W
              Wayne Workman @JGallo
              last edited by Oct 19, 2017, 5:48 PM

              @jgallo So, rebooting after you clear the cache nullifies the effects that clearing the cache does, because rebooting does the same thing. I’m trying to isolate if clearing the cache fixes the problems for you.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
              Daily Clean Installation Results:
              https://fogtesting.fogproject.us/
              FOG Reporting:
              https://fog-external-reporting-results.fogproject.us/

              J 3 Replies Last reply Oct 19, 2017, 5:51 PM Reply Quote 0
              • J
                JGallo @Wayne Workman
                last edited by Oct 19, 2017, 5:51 PM

                @wayne-workman

                ahh ok. I figured rebooting would reboot all services. Maybe the storage nodes needed a reboot this whole time LOL. There were a few that have been on non-stop for about 90 days. If the messages return, I will do a clear cache and not reboot.

                1 Reply Last reply Reply Quote 0
                • J
                  JGallo @Wayne Workman
                  last edited by Oct 19, 2017, 10:17 PM

                  @wayne-workman

                  So after some time again message came back even after clearing cache and not rebooting. We had to image a computer and I noticed that when the computer tried to check in after the iPXE files were loading prior to imaging that the error for the computer checking in stated the same message as the storage nodes on the dashboard valid database connection could be made. I will image another computer and screenshot that error.

                  1 Reply Last reply Reply Quote 0
                  • J
                    JGallo @Wayne Workman
                    last edited by Oct 20, 2017, 3:30 PM

                    @wayne-workman

                    Sorry about the late reply here. We had a rogue router on a campus which we were hunting for and eventually found. The error message about the database connection after host registration and prior to actually imaging disappeared. Going to restart again and continue to monitor issue.

                    W 1 Reply Last reply Oct 20, 2017, 6:26 PM Reply Quote 0
                    • W
                      Wayne Workman @JGallo
                      last edited by Oct 20, 2017, 6:26 PM

                      @jgallo That makes a lot of sense now actually - because nothing about this problem was making sense before. I guess there were IP conflicts on your network causing all the issues - or possibly a routing loop.

                      Glad it seems to be fixed - let us know if it’s not.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                      Daily Clean Installation Results:
                      https://fogtesting.fogproject.us/
                      FOG Reporting:
                      https://fog-external-reporting-results.fogproject.us/

                      J 2 Replies Last reply Oct 20, 2017, 6:51 PM Reply Quote 0
                      • J
                        JGallo @Wayne Workman
                        last edited by Oct 20, 2017, 6:51 PM

                        @wayne-workman

                        Yeh these high schoolers get curious and it wouldn’t be the first time we have dealt with rogue routers.

                        1 Reply Last reply Reply Quote 0
                        • J
                          JGallo @Wayne Workman
                          last edited by Oct 31, 2017, 6:22 PM

                          @wayne-workman

                          So after several different attempts to see this issue be resolved, the issue continues to persist. The patterns I have observed is that it now takes about an hour for the issue to begin. I have gone into the /etc/mysql/mysql.conf.d/mysqld.cnf file and edited the bind address to each node because on a seperate post it was recommended to use the IP of the node instead of the loopback. Don’t think this worked because I restarted server and node after these changes and problems persist. I also went into the /var/www/fog/lib/fog/fogurlrequests.class.php and edited the aconntimeout = 2000 to 10000 as well as the counttimeout = 15; to 30. Would it be possible @Tom-Elliott or someone to remote in to check this out remotely? I know all the passwords for fogstorage accounts are accurate because imaging works fine. Registering computers at sites works fine as well as image replication to storage groups work fine. I could start from scratch and add one node at a time but I could not be able to place a test fog server in place due to this being a production FOG system in place. I have back ups of the images so starting from scratch is not a problem. I have a strange feeling this could be related to Ubuntu and updates with PHP7. Thank you

                          J 1 Reply Last reply Oct 31, 2017, 6:38 PM Reply Quote 0
                          • J
                            JGallo @JGallo
                            last edited by Oct 31, 2017, 6:38 PM

                            Another observation is that upon the sudo service mysql restart on the fog server, the error goes away for DefaultMember in the dashboard. I tried the same thing on other storage nodes but the message is still in the dashboard. Maybe this could be something with remote connections with the mysql database.

                            1 Reply Last reply Reply Quote 0
                            • J
                              JGallo
                              last edited by Oct 31, 2017, 8:38 PM

                              @Wayne-Workman

                              @Sebastian-Roth remoted in and looks like mysql was limited to 150 connections. He raised it to 250 and now it would appear fixed so far. I’m keeping an eye on the max used connections and it hasn’t gone over 161 so I must have been hitting that 150 max. Thanks for all your help here. Greatly appreciate it.

                              1 Reply Last reply Reply Quote 1
                              • S
                                Sebastian Roth Moderator
                                last edited by Oct 31, 2017, 8:40 PM

                                Turns out mysql hit the max connection limit. Not sure what the default limit is but on this system it was 151 concurrent connections. Check for connections numbers are just an example - where much higher on the actual system):

                                mysql> show status like '%onn%';
                                +----------------------------------+-------+
                                | Variable_name                    | Value |
                                +----------------------------------+-------+
                                | Aborted_connects                 | 0     | 
                                | Connections                      | 8     | 
                                | Max_used_connections             | 4     | 
                                | Connection_error_max_connections | 1421  |
                                | Ssl_client_connects              | 0     | 
                                | Ssl_connect_renegotiates         | 0     | 
                                | Ssl_finished_connects            | 0     | 
                                | Threads_connected                | 4     | 
                                +----------------------------------+-------+
                                

                                See that Connection_error_max_connections count. So we checked the max_connections setting and increased it:

                                mysql> show variables like 'max_connections';
                                +-----------------+-------+
                                | Variable_name   | Value |
                                +-----------------+-------+
                                | max_connections | 151   |
                                +-----------------+-------+
                                mysql> set global max_connections = 250;
                                

                                The change takes effect without restarting the server or mysql!

                                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                J 1 Reply Last reply Nov 2, 2017, 6:47 PM Reply Quote 1
                                • J
                                  JGallo @Sebastian Roth
                                  last edited by Nov 2, 2017, 6:47 PM

                                  @sebastian-roth

                                  I did a git pull and updated FOG to address a Windows 10 issue and upon the update the issue I had with database connections returned. It would appear that mysql goes back to default max connections of 151 after the update. Does the fog installer/updater have mysql update settings? If it does, could I suggest an increase on that max connection limit be part of the fog installer/updater? Or is this something that needs to be manually entered in mysql with the amount of storage nodes and groups I have which was done already to address my issue. Thank you again for your help on this.

                                  W 1 Reply Last reply Nov 2, 2017, 10:26 PM Reply Quote 0
                                  • W
                                    Wayne Workman @JGallo
                                    last edited by Wayne Workman Nov 2, 2017, 4:26 PM Nov 2, 2017, 10:26 PM

                                    @jgallo said in FOG Web GUI speed and default storage activity:

                                    It would appear that mysql goes back to default max connections of 151 after the update.

                                    Because the setting was only changed in the MySQL config live. The setting has to be set in the mysql configuration file using the process outlined here: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_MySQL#Increase_maximum_simultaneous_MySQL_connections

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                    Daily Clean Installation Results:
                                    https://fogtesting.fogproject.us/
                                    FOG Reporting:
                                    https://fog-external-reporting-results.fogproject.us/

                                    J 1 Reply Last reply Nov 2, 2017, 10:53 PM Reply Quote 0
                                    • J
                                      JGallo @Wayne Workman
                                      last edited by JGallo Nov 2, 2017, 4:54 PM Nov 2, 2017, 10:53 PM

                                      @wayne-workman

                                      I’m following the steps in Wiki but mysql is on Ubuntu 16.04.3 LTS and when I go to /etc/mysql/my.cnf and try to add the increase of max connections, it errors out. Modifying the mysqld.cnf in /etc/mysql/mysql.conf.d and removing the comment on the line with max connections works. However it’s really odd that when you restart mysql and go into mysql to check the max connections it would show a smaller value then defined in the mysqld.cnf file. Am I right to modify the mysqld.cnf? I’m lost at this but I know that at least the value is increased so it works as intended but with the incorrect value I defined.

                                      W 1 Reply Last reply Nov 3, 2017, 12:13 AM Reply Quote 0
                                      • W
                                        Wayne Workman @JGallo
                                        last edited by Nov 3, 2017, 12:13 AM

                                        @jgallo I’ll have to look into it, but mysql and mariadb have configuration files where this setting can be used - and when you restart mysql/mariadb it should use the setting and work.

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                        Daily Clean Installation Results:
                                        https://fogtesting.fogproject.us/
                                        FOG Reporting:
                                        https://fog-external-reporting-results.fogproject.us/

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          Sebastian Roth Moderator
                                          last edited by Nov 3, 2017, 4:24 AM

                                          @JGallo @Wayne-Workman Just found this on stackexchange. Though it doesn’t say so anywhere in the mysql manual it’s possible that mysql does somehow restrict the max_connections when it’s higher than some internal memory calculations. Maybe give the mysql mem calc a try. On the other hand setting the value up to 250 in the live DB worked like a charm.

                                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                          W 1 Reply Last reply Nov 3, 2017, 1:03 PM Reply Quote 0
                                          • 1
                                          • 2
                                          • 3
                                          • 4
                                          • 3 / 4
                                          • First post
                                            Last post

                                          219

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project