• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    SVN 4972 to SVN 5046 server load

    Scheduled Pinned Locked Moved Solved
    FOG Problems
    5
    28
    11.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      Sebastian Roth Moderator
      last edited by

      How many apache processed do you see running ps ax | grep apache | wc -l ?

      And what about the connections lsof -i :80 ?

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      1 Reply Last reply Reply Quote 0
      • Joseph HalesJ
        Joseph Hales Testers
        last edited by

        189 processes
        lsof output.txt

        RTFM

        1 Reply Last reply Reply Quote 0
        • Joseph HalesJ
          Joseph Hales Testers
          last edited by

          Updated to svn 5020 server load is averaging 32 now.

          RTFM

          1 Reply Last reply Reply Quote 0
          • george1421G
            george1421 Moderator
            last edited by

            What does top show you? That will tell you what is consuming your clock cycles.

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

            Wayne WorkmanW 1 Reply Last reply Reply Quote 1
            • Wayne WorkmanW
              Wayne Workman @george1421
              last edited by Wayne Workman

              @george1421 Likely a ton of apache stuff… it comes down to how the fog hosts/fog clients interacts regularly with the web interface for instructions…

              @Developers it might be worth exploring some sort of other way to provide data to the hosts.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
              Daily Clean Installation Results:
              https://fogtesting.fogproject.us/
              FOG Reporting:
              https://fog-external-reporting-results.fogproject.us/

              1 Reply Last reply Reply Quote 0
              • Joseph HalesJ
                Joseph Hales Testers
                last edited by Joseph Hales

                top - 14:17:35 up  3:08,  1 user,  load average: 33.54, 34.93, 33.54
                Tasks: 465 total,  22 running, 443 sleeping,   0 stopped,   0 zombie
                %Cpu(s): 35.5 us,  4.3 sy,  0.0 ni, 44.0 id, 15.7 wa,  0.0 hi,  0.6 si,  0.0 st
                KiB Mem:  24678932 total,  9619308 used, 15059624 free,   122692 buffers
                KiB Swap:  4844540 total,        0 used,  4844540 free.  7271580 cached Mem
                
                  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                 1398 mysql     20   0 8730436 336472   7328 S  31.2  1.4 115:48.88 mysqld
                14010 www-data  20   0  333584  23144   7052 R  26.2  0.1   0:09.48 apache2
                13967 www-data  20   0  327440  16604   5688 S  22.5  0.1   0:11.39 apache2
                14066 www-data  20   0  327216  17696   7016 S  16.2  0.1   0:11.28 apache2
                13942 www-data  20   0  327184  16116   5456 S  15.9  0.1   0:12.02 apache2
                12817 www-data  20   0  333356  22988   7260 R  14.9  0.1   1:13.09 apache2
                14045 www-data  20   0  327180  16100   5444 S  14.9  0.1   0:07.29 apache2
                13317 www-data  20   0  333616  21536   5724 S  14.6  0.1   0:40.15 apache2
                13991 www-data  20   0  333328  21088   5424 R  14.6  0.1   0:08.65 apache2
                12793 www-data  20   0  333356  21516   5772 S  14.3  0.1   1:19.05 apache2
                13996 www-data  20   0  333324  21148   5384 S  14.3  0.1   0:08.57 apache2
                12825 www-data  20   0  333392  22816   7076 R  13.9  0.1   1:24.51 apache2
                14211 www-data  20   0  328404  16856   5280 S  13.9  0.1   0:00.42 apache2
                13997 www-data  20   0  327180  17644   6984 S  10.9  0.1   0:18.68 apache2
                14128 www-data  20   0  327184  16040   5384 S  10.6  0.1   0:04.77 apache2
                14171 www-data  20   0  327180  16040   5384 S  10.3  0.1   0:03.64 apache2
                14054 www-data  20   0  327216  16132   5444 S   9.9  0.1   0:08.68 apache2```

                RTFM

                1 Reply Last reply Reply Quote 0
                • george1421G
                  george1421 Moderator
                  last edited by

                  From the top output it does show you have a lot of apache stuff and they are all doing something since it also appears that msyql is running pretty hard too. So your system IS doing something.

                  What is going on at the time this snapshot was run? Were you deploying? Do you have a large campus where you have a lot of devices talking back to the fog server?

                  /var/log/httpd/access_log and /var/log/httpd/error_log might give insight on what is hitting your apache server.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                  1 Reply Last reply Reply Quote 0
                  • Tom ElliottT
                    Tom Elliott
                    last edited by

                    Can you please try reupdating? I removed some destruct code a while back and based on what it sounds like is happening is the links are completing but not releasing the links. Maybe with it back all will be better?

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    1 Reply Last reply Reply Quote 0
                    • Joseph HalesJ
                      Joseph Hales Testers
                      last edited by

                      I have one of the largest deployments running SVN but like I said this was fine under a previous SVN one of the earlier changes that was made was to remove some sql validation checks that were used previously to clean incomplete records from the db it seems innodb didn’t like the function calls I wonder if these have been readded since?

                      RTFM

                      Tom ElliottT 1 Reply Last reply Reply Quote 0
                      • Tom ElliottT
                        Tom Elliott @Joseph Hales
                        last edited by

                        @Joseph-Hales I have not added back any invalid entry checks.

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        1 Reply Last reply Reply Quote 0
                        • S
                          Sebastian Roth Moderator
                          last edited by

                          @Joseph-Hales Thank you so much for the lsof output text file. I feel like this is kind of proofing that it’s the clients causing this massive storm of apache childs and probably mysql load as well. This is why Tom and I are a bit helpless debugging this issue as we don’t have such a big test environment. I just grepped through the output and found roughly 120 established connections to distinct hosts/clients. And another 50 or so in FIN_WAIT state.

                          I think Tom is pretty convinced that it is something he’s done in the code. Maybe it is but I still wonder why clients keep open connections and even don’t close them properly. Maybe this was introduced with the new client? Sorry if I am heading the wrong way here. It’s just me guessing…

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          1 Reply Last reply Reply Quote 0
                          • Joseph HalesJ
                            Joseph Hales Testers
                            last edited by

                            I am only running the legacy client on my hosts.

                            RTFM

                            1 Reply Last reply Reply Quote 0
                            • Joseph HalesJ
                              Joseph Hales Testers
                              last edited by

                              After updating SVN to 5054 and restarting the server completly it still seems high to me.

                              System Overview
                              Username	jhales
                              Web Server	10.200.10.150
                              TFTP Server	10.200.10.150
                              Load Average	29.00, 31.61, 20.87
                              System Uptime	13 min, 0 users```

                              RTFM

                              1 Reply Last reply Reply Quote 0
                              • Joseph HalesJ
                                Joseph Hales Testers
                                last edited by

                                FYI I can load the pages but they are extremely slow and sometimes fail to load at all.

                                RTFM

                                1 Reply Last reply Reply Quote 0
                                • S
                                  Sebastian Roth Moderator
                                  last edited by

                                  @Joseph-Hales said:

                                  I am only running the legacy client on my hosts.

                                  Thanks for clarifying!! So this idea was really a dead end. That’s good news as I think we can rule out the possibility of it being an issue introduced by the client. Must be the PHP code then I reckon.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  1 Reply Last reply Reply Quote 0
                                  • Wayne WorkmanW
                                    Wayne Workman
                                    last edited by Wayne Workman

                                    How about making the checkin time even greater?

                                    I mean, there is no reason for 10,000 hosts in the middle of the school year to be checking for tasks every 2 minutes… that’s kind of absurd.

                                    Why not set it to 2 hours?

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                    Daily Clean Installation Results:
                                    https://fogtesting.fogproject.us/
                                    FOG Reporting:
                                    https://fog-external-reporting-results.fogproject.us/

                                    1 Reply Last reply Reply Quote 0
                                    • Joseph HalesJ
                                      Joseph Hales Testers
                                      last edited by

                                      Do they still checkin on boot? The reason I ask is that I would like a newly imaged machine to pull snapins as soon as possible.

                                      RTFM

                                      Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                                      • Joseph HalesJ
                                        Joseph Hales Testers
                                        last edited by

                                        SVN 5058 fixes the main issue or at least makes it manageable.

                                        Username	jhales
                                        Web Server	10.200.10.150
                                        TFTP Server	10.200.10.150
                                        Load Average	6.79, 13.52, 13.55
                                        System Uptime	1:28, 1 user
                                        

                                        RTFM

                                        1 Reply Last reply Reply Quote 1
                                        • Wayne WorkmanW
                                          Wayne Workman @Joseph Hales
                                          last edited by

                                          @Joseph-Hales Not sure…

                                          but is it worth 10,000 hosts hitting one server every few minutes just so the occasional imaged computer gets snapins fast? To me it’s not worth it.

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                          Daily Clean Installation Results:
                                          https://fogtesting.fogproject.us/
                                          FOG Reporting:
                                          https://fog-external-reporting-results.fogproject.us/

                                          1 Reply Last reply Reply Quote 0
                                          • Joseph HalesJ
                                            Joseph Hales Testers
                                            last edited by

                                            we image 20 to 50 pc’s a day on average.

                                            RTFM

                                            Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post

                                            169

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project