• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    FOG Unresponsive under "heavy" load

    Scheduled Pinned Locked Moved Unsolved
    FOG Problems
    6
    36
    6.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • fry_pF
      fry_p Moderator @george1421
      last edited by

      @george1421 20 unicast deployments. I also seemed to have triggered it when truncating the imaging log in mysql.

      Like open source community computing? Why not do it for a good cause?
      Use your computer/server for humanitarian projects when it is idle!
      https://join.worldcommunitygrid.org?recruiterId=1026912

      george1421G 1 Reply Last reply Reply Quote 0
      • george1421G
        george1421 Moderator @fry_p
        last edited by george1421

        @fry_p well I have no proof of this but my intuition is telling me that php-fpm is probably running out of memory when you are unicasting to that many systems. So as an experiment I want you to do this .

        1. We need to locate a file called www.conf in the etc directory. It should be in a directory that has php-fpm in the path. Use this command.
          find /etc -name www.conf
        2. Edit that file down towards the bottom. You should see a section that has a few entries that start out with php_admin_value. I want you to add a new line with this:
          php_admin_value[memory_limit] = 256M The exact placement of the line doesn’t really matter but keep it in the admin value section.
        3. Save and exit your text editor.
        4. Restart pgp-fpm and apache (make sure you don’t have imaging running when you do this)
        sudo systemctl restart php-fpm
        sudo systemctl restart apache2
        

        Now when you have time or your next be image push see if you run into issue again.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        fry_pF L 3 Replies Last reply Reply Quote 2
        • fry_pF
          fry_p Moderator @george1421
          last edited by

          @george1421 We will be doing an exorbitant of lab re-imaging this summer, so testing will not be an issue. I will do this and certainly report back. Probably in the next few days.

          Like open source community computing? Why not do it for a good cause?
          Use your computer/server for humanitarian projects when it is idle!
          https://join.worldcommunitygrid.org?recruiterId=1026912

          1 Reply Last reply Reply Quote 1
          • fry_pF
            fry_p Moderator @george1421
            last edited by fry_p

            @george1421 I made the changes but it got me thinking. I’ve been meaning to rebuild our FOG server on a more proper OS (Centos 7), so on Friday night, I did just that. I’ll let you know if I have any issues with mass unicasting now, but the variables have changed. I feel a lot better about the stability with the new install for now.

            Like open source community computing? Why not do it for a good cause?
            Use your computer/server for humanitarian projects when it is idle!
            https://join.worldcommunitygrid.org?recruiterId=1026912

            1 Reply Last reply Reply Quote 0
            • Q
              Quazz Moderator
              last edited by Quazz

              I’m wondering if the Ondemand FPM handler is the better choice for FOG in general and such cases specifically.

              In my experience Ondemand is only marginally slower than Dynamic or Static, but uses far less RAM on average. It’s also far easier to setup correctly since you don’t require minimum children or anything like that.

              The problem with the current set up is that FPM processes that have claimed a lot of RAM will only respawn after they’ve met the request limit which could take ages in certain scenarios.

              In Ondemand you can specify the idle timeout so that if a process is doing nothing it will be killed off and the memory freed to the system.

              I will also recommend the Event MPM for Apache alongside this. There is little point to remain with Prefork when we are using FPM anyway.

              1 Reply Last reply Reply Quote 0
              • L
                LibraryMark @george1421
                last edited by LibraryMark

                I am running Ubuntu 16.04 server in vmware. I was never able to make 1.5.4 multicast until I made the changes outlined here to the www.conf file. I was suffering the same things that fry_p had probloms with. Downloading boot.php would just be “…” for days. Now it works (like it used to).

                Thank you, george1421!

                Edit: Well, I take that back (a little bit). After a 20-pc multicast session, none of the PC’s were able to ‘update the database’. I had to cancel the session, reboot the fog server, and reboot the PC’s . At least the image was successfully blasted out otherwise I would be having a bad day right about now.

                L george1421G 2 Replies Last reply Reply Quote 0
                • L
                  LibraryMark @LibraryMark
                  last edited by

                  @librarymark
                  And while trying to multicast 8 pcs, now I get this again: alt text

                  L Q 2 Replies Last reply Reply Quote 0
                  • L
                    LibraryMark @LibraryMark
                    last edited by

                    @librarymark
                    And after I reboot the server and the multicast actually runs, the PC’s are stuck at this:
                    0_1528808253831_295033fa-25d0-497e-8c95-59aef6f22f3a-image.png

                    and FOG’s webpage says this:
                    0_1528808342767_b826c179-b290-4e0e-842f-42e53a8d96b9-image.png

                    george1421G 2 Replies Last reply Reply Quote 0
                    • Q
                      Quazz Moderator @LibraryMark
                      last edited by

                      @librarymark Do you get php memory exhaustion in the logs?

                      Would also be interested in seeing your free -m and top (shift+m) stats when this happens.

                      1 Reply Last reply Reply Quote 0
                      • george1421G
                        george1421 Moderator @LibraryMark
                        last edited by

                        @librarymark said in FOG Unresponsive under "heavy" load:

                        Edit: Well, I take that back (a little bit). After a 20-pc multicast session, none of the PC’s were able to ‘update the database’. I had to cancel the session, reboot the fog server, and reboot the PC’s . At least the image was successfully blasted out otherwise I would be having a bad day right about now.

                        It would be interesting to know the memory usage when this broke.

                        Also just for clarity what updates did you do to the www.conf file, up the memory to 256MB?

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                        L 1 Reply Last reply Reply Quote 0
                        • george1421G
                          george1421 Moderator @LibraryMark
                          last edited by

                          @librarymark OK for the gateway timeout lets work with that. I think if you look in the apache error file. You will see a php timeout waiting for php-fpm to respond. What we need to do is tell apache to wait a bit before timing out.

                          About how long does it take to push out your image to 20 computers?

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                          1 Reply Last reply Reply Quote 0
                          • L
                            LibraryMark @george1421
                            last edited by LibraryMark

                            @george1421
                            I just upped the memory in/etc/php/7.1/fpm/pool.d/www.conf:

                            php_admin_value[memory_limit] = 256M 
                            
                            1 Reply Last reply Reply Quote 1
                            • george1421G
                              george1421 Moderator @LibraryMark
                              last edited by george1421

                              @librarymark What I want you to test is outlined in this post: https://forums.fogproject.org/topic/11713/503-service-unavailable-error/40

                              I want you to update this section

                                  <Proxy "fcgi://127.0.0.1:9000">
                                      ProxySet timeout=500
                                  </Proxy>
                              

                              Set the timeout in seconds to be just a bit longer than your push time.

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                              L 1 Reply Last reply Reply Quote 0
                              • L
                                LibraryMark @george1421
                                last edited by LibraryMark

                                @george1421

                                Where do I find the “push time”?

                                I edited the file is /etc/apache2/sites-enabled/001-fog.conf, and it now looks like this:

                                <VirtualHost *:80>
                                  <Proxy "fcgi://127.0.0.1:9000">
                                        ProxySet timeout=300
                                   </Proxy>
                                
                                    <FilesMatch "\.php$">
                                        SetHandler "proxy:fcgi://127.0.0.1:9000/"
                                    </FilesMatch>
                                    KeepAlive Off
                                    ServerName 10.5.0.61
                                    DocumentRoot /var/www/html/
                                    <Directory /var/www/html/fog/>
                                        DirectoryIndex index.php index.html index.htm
                                    </Directory>
                                    RewriteEngine On
                                    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
                                    RewriteRule .* - [F]
                                    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
                                    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-d
                                    RewriteRule ^/fog/(.*)$ /fog/api/index.php [QSA,L]
                                </VirtualHost>
                                
                                

                                Is that correct? In any case I will not be able to test now because we just opened (public library). It might be a few days.

                                george1421G 1 Reply Last reply Reply Quote 0
                                • george1421G
                                  george1421 Moderator @LibraryMark
                                  last edited by george1421

                                  @librarymark Right that looks good. Make sure you set the timeout to the right number of seconds. Right now as configured apache will wait 5 minutes for php-fpm to respond before giving up. If your image push time is more than 5 minutes you need to adjust this number.

                                  [edit] Sorry, I was not clear “push time” is the time it takes to send the image to all 20 computers when using a multicast image.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                  L 1 Reply Last reply Reply Quote 0
                                  • L
                                    LibraryMark @george1421
                                    last edited by

                                    @george1421
                                    My multicast sessions usually take about 5-7 minutes to complete. Is that what you mean?

                                    george1421G 1 Reply Last reply Reply Quote 0
                                    • george1421G
                                      george1421 Moderator @LibraryMark
                                      last edited by

                                      @librarymark yes that is what I mean. So change that 300 to 500. Understand this is only a test to see if that IS what is causing your multicast failure. If this proves out we will need to get back with the developers to understand why?

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                      L 1 Reply Last reply Reply Quote 0
                                      • L
                                        LibraryMark @george1421
                                        last edited by LibraryMark

                                        @george1421
                                        How many versions back do we need to go to bypass this? I think I want to step back in time until the bugs are worked out. Or - what older version of Ubuntu would work better?

                                        george1421G 1 Reply Last reply Reply Quote 0
                                        • george1421G
                                          george1421 Moderator @LibraryMark
                                          last edited by

                                          @librarymark Well if its a php-fpm issue, you can just turn it off and bypass the issue.

                                          The down side is if we don’t have different use cases helping us to understand what is going wrong it will never get fixed. But if you want to disable php-fpm I can tell you how its just adding one line to the fog.conf file for apache.

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                          L 1 Reply Last reply Reply Quote 0
                                          • L
                                            LibraryMark @george1421
                                            last edited by LibraryMark

                                            @george1421
                                            I am curious what php-fpm gains us. I have never had performance issues before.

                                            george1421G 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post

                                            203

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project