• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

Fog using a huge amount of memory and failing

Scheduled Pinned Locked Moved Unsolved
FOG Problems
5
27
3.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J
    jstephen
    last edited by Jun 6, 2018, 12:31 PM

    I’ve been seeing a lot of sluggishness in the web UI for the past several updates, but didn’t think much of it. I updated to 1.5.3 earlier this week, and the memory usage from Fog is huge, and seems to now be effecting not just the web UI but also the base Fog functions - imaging is slower, tftp connections are timing out, etc.

    I’ve assigned the Debian virtual machine that is running Fog 24576 MB of memory, and even with that much it is still trying to use more. Looking in top, apache2 has dozens of processes running and is sucking up all the memory. This is just when the server is sitting idle.

    Any suggestions on how to troubleshoot this problem?

    1 Reply Last reply Reply Quote 0
    • G
      george1421 Moderator
      last edited by Jun 6, 2018, 12:35 PM

      Please upgrade up to 1.5.4. There has been several fixes that will impact the memory usage (in a good way).

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

      1 Reply Last reply Reply Quote 0
      • J
        jstephen
        last edited by Jun 6, 2018, 1:14 PM

        OK, I’ve updated to 1.5.4, and I’d say it is at least marginally better. In the web interface, I tried opening 10 host pages, and it immediately bogged down again and wouldn’t load anything for a minute or two.

        I can’t really say whether the problem is resolved yet since it always runs a bit better after restarting apache.

        I’ll check it throughout the day and post back if there are continued problems.

        Thanks!

        G 1 Reply Last reply Jun 6, 2018, 1:28 PM Reply Quote 0
        • G
          george1421 Moderator @jstephen
          last edited by Jun 6, 2018, 1:28 PM

          @jstephen How much vMemory do you have on this VM as well as the vCPU?

          Lastly how many fog clients connect to this fog server?

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

          J 1 Reply Last reply Jun 6, 2018, 3:11 PM Reply Quote 0
          • Q
            Quazz Moderator
            last edited by Quazz Jun 6, 2018, 8:47 AM Jun 6, 2018, 2:45 PM

            Is that 24.5GB of RAM?

            And apache2 is using most of it?

            What about php or php-fpm?

            Did you manually change some of the apache settings?

            What does apache2ctl -l show?

            J 1 Reply Last reply Jun 6, 2018, 3:54 PM Reply Quote 1
            • J
              jstephen @george1421
              last edited by Jun 6, 2018, 3:11 PM

              @george1421 It has dynamic memory in a range from 4096 MB - 24576 MB.

              For the vCPU, I just went with the defaults from Hyper-V I think - this was setup about 4 years ago. The description in the settings says:
              Processors: 1
              NUMA nodes: 1
              Sockets: 1
              Numa Topology:
              Maximum number of processors: 12
              Maximim amount of memory 13348 MB

              We’ve got right around 500 clients.

              1 Reply Last reply Reply Quote 0
              • J
                jstephen @Quazz
                last edited by Jun 6, 2018, 3:54 PM

                @quazz Yes, 24.5GB of RAM.

                Sitting idle, there are 25 apache2 processes using a total of 17% of the available memory. Apache is also using a lot of the cpu percentage as well - 64%.

                php-fpm also has about 3 processes, but looks to be taking up only about 1.5% of the RAM.

                I think a few things have likely been changed in the apache settings. The server was upgraded from Debian 7 last year which necessitated a few changes I think.

                apache2ctl -l shows: core.c, mod_so.c, mod_watchdog.c, http_core.c, mod_log_config.c, mod_logio.c mod_version.c, and mod_unixd.c

                W G 2 Replies Last reply Jun 6, 2018, 5:27 PM Reply Quote 0
                • W
                  Wayne Workman @jstephen
                  last edited by Wayne Workman Jun 6, 2018, 11:28 AM Jun 6, 2018, 5:27 PM

                  @jstephen Can you add this to your apache configuration file and see if the issue resolves itself:

                  MaxRequestWorkers 100
                  MaxConnectionsPerChild 25
                  

                  You’ll need to restart apache after adding this.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                  Daily Clean Installation Results:
                  https://fogtesting.fogproject.us/
                  FOG Reporting:
                  https://fog-external-reporting-results.fogproject.us/

                  J 1 Reply Last reply Jun 7, 2018, 12:10 PM Reply Quote 0
                  • J
                    jstephen @Wayne Workman
                    last edited by Jun 7, 2018, 12:10 PM

                    @wayne-workman I added those to mpm_event.conf which seemed like the right place to put them, but I’m not entirely sure!

                    I can’t really see any difference… maybe it is a tad more stable? What I’ve noticed more than anything with this testing though is that when I restart apache, it runs way, way better for a while. Like this morning, each page in the Fog UI was taking about a minute to load; I restarted apache and it was down to 5 seconds per page even when opening multiple pages at once.

                    W 1 Reply Last reply Jun 7, 2018, 1:32 PM Reply Quote 0
                    • W
                      Wayne Workman @jstephen
                      last edited by Wayne Workman Jun 7, 2018, 7:33 AM Jun 7, 2018, 1:32 PM

                      @jstephen Try adding the configuration lines to the bottom of this file: /etc/apache2/apache2.conf and then restart apache with systemctl restart apache2 and see what happens. Also, how many cores does your fog server have?

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                      Daily Clean Installation Results:
                      https://fogtesting.fogproject.us/
                      FOG Reporting:
                      https://fog-external-reporting-results.fogproject.us/

                      1 Reply Last reply Reply Quote 0
                      • G
                        george1421 Moderator @jstephen
                        last edited by Jun 7, 2018, 3:52 PM

                        @jstephen said in Fog using a huge amount of memory and failing:

                        Yes, 24.5GB of RAM.
                        Sitting idle, there are 25 apache2 processes using a total of 17% of the available memory. Apache is also using a lot of the cpu percentage as well - 64%.
                        php-fpm also has about 3 processes, but looks to be taking up only about 1.5% of the RAM.

                        There has to be something else going on here. These settings are a bit out of wack.

                        First on the questions I asked you. You might consider 2 vCPUs at 4GB of ram. That is all FOG should use. If you have a 1000 or more hosts then we might need to bump up the numbers a bit.

                        With FOG 1.5.1 and later FOG now uses pgp-fpm for its pgp engine and moved away from apache’s built in php engine. This should lower the cpu requirements of apache by off-loading php requests to php-fpm. With apache using the majority of the CPU still, it almost sounds like the php activities have not shifted to php-fpm.

                        What I would like you to do is create a file in the root of the web server directory, typically /var/www/html name that file info.php and make it world readable. In that file place the following code:

                        <html>
                        <body>
                        
                        <?php
                         phpinfo();
                        ?>
                        
                        </body>
                        </html>
                        

                        Once its created navigate to that file by using this url http://<fog_server_ip>/info.php

                        I’m expecting to see a this field set to
                        0_1528386629888_php-fpm.png
                        and not this
                        0_1528386636519_php-apache.png

                        The first screen shows that php-fpm is managing the php process. The second screen is the way fog was setup before version 1.5.1. I would expect php-fpm to have the workload here, not apache.

                        ref: https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                        J J 2 Replies Last reply Jun 7, 2018, 4:12 PM Reply Quote 0
                        • J
                          Joe Gill @george1421
                          last edited by Jun 7, 2018, 4:12 PM

                          @george1421
                          I have also been experiencing high memory usage and sluggishness. I completely lost my interface and had to restart PHP in order for the interface to return. I will also try your suggestion here to see what is going on.

                          G 1 Reply Last reply Jun 7, 2018, 4:14 PM Reply Quote 0
                          • G
                            george1421 Moderator @Joe Gill
                            last edited by Jun 7, 2018, 4:14 PM

                            @joe-gill If you are not on FOG 1.5.4, but are using 1.5.1 to 1.5.3 there is a fix that is needed to php-fpm to address memory bloat under certain conditions.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                            J 2 Replies Last reply Jun 7, 2018, 4:15 PM Reply Quote 0
                            • J
                              Joe Gill @george1421
                              last edited by Jun 7, 2018, 4:15 PM

                              @george1421
                              I actually upgraded to FOG 1.5.4 and the issue persisted.

                              G 1 Reply Last reply Jun 7, 2018, 4:18 PM Reply Quote 0
                              • G
                                george1421 Moderator @Joe Gill
                                last edited by george1421 Jun 7, 2018, 11:19 AM Jun 7, 2018, 4:18 PM

                                @joe-gill ok we need to collect a bit more info here. It sounds like a trend is starting.

                                1. What OS is the host OS for FOG?
                                2. What is your memory installed in the fog server?
                                3. What process is using the most memory?
                                4. What process is using the most CPU utilization?
                                5. Number of clients you have that has fog installed
                                6. Fog check in interval Fog Configuration -> Fog Settings -> Fog Client -> FOG_CLIENT_CHECKIN_TIME (if you have less than 300 [5 minutes] you might want to bump that up to 300 or 600 seconds).

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                J 1 Reply Last reply Jun 7, 2018, 5:07 PM Reply Quote 0
                                • J
                                  Joe Gill @george1421
                                  last edited by Jun 7, 2018, 4:44 PM

                                  @george1421

                                  Let me know if their is anything else you would like me to check. Thanks!

                                  0_1528389858393_FOG_PHP.png

                                  G 1 Reply Last reply Jun 7, 2018, 4:52 PM Reply Quote 0
                                  • G
                                    george1421 Moderator @Joe Gill
                                    last edited by george1421 Jun 7, 2018, 10:52 AM Jun 7, 2018, 4:52 PM

                                    @joe-gill Ok you are using php-fpm. That is good. Now we need to understand what is consuming the memory. I had a previous post where I was asking specific questions about your current setup. Please respond to that… and

                                    Depending on your distro there is a php-fpm file that fog creates it should be in /etc (in a sub directory) called www.conf. Under centos 7 its in /etc/php-fpm.d Lets review that file, towards the middle there should be this line uncommented.

                                    pm.max_requests = 2000
                                    

                                    Let make sure its not commented out.

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                    1 Reply Last reply Reply Quote 0
                                    • J
                                      Joe Gill @george1421
                                      last edited by Jun 7, 2018, 5:07 PM

                                      @george1421
                                      Running on a Hyper V Virtual Machine
                                      FOG OS - Debian 9.4
                                      Memory - 4 GB
                                      CPU - 2 CPUs allotted

                                      I used Top to determine these. If their is a better command let me know and I will certainly run that and report back.

                                      Memory Usage (in order from most to least) –
                                      Apache2 (20 percent)
                                      mysql
                                      FOGImageSize
                                      FOGPingHosts
                                      FOGSnapinHash

                                      CPU Usage –
                                      Apache2

                                      G 1 Reply Last reply Jun 7, 2018, 5:15 PM Reply Quote 0
                                      • G
                                        george1421 Moderator @Joe Gill
                                        last edited by george1421 Jun 7, 2018, 11:19 AM Jun 7, 2018, 5:15 PM

                                        @joe-gill Can I get to bits of info I forgot to ask?

                                        1. Top CPU usage service
                                        2. Number of clients you have in your FOG environment
                                        3. Fog check in interval Fog Configuration -> Fog Settings -> Fog Client -> FOG_CLIENT_CHECKIN_TIME (if you have less than 300 [5 minutes] you might want to bump that up to 300 or 600 seconds).

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                        J 1 Reply Last reply Jun 7, 2018, 5:19 PM Reply Quote 0
                                        • J
                                          Joe Gill @george1421
                                          last edited by Jun 7, 2018, 5:19 PM

                                          @george1421
                                          Top CPU usage service is Apache2.
                                          Total hosts is ~500.

                                          Where is the www.conf file located in Debian? I searched high and low and could not find it. I found something in the /etc/php/7.0/ directory but it didn’t have pm.max_requests.

                                          Thanks!

                                          G 1 Reply Last reply Jun 7, 2018, 5:21 PM Reply Quote 0
                                          • 1
                                          • 2
                                          • 2 / 2
                                          2 / 2
                                          • First post
                                            4/27
                                            Last post

                                          188

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project