• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Fog SVN 5020 and above CPU Hammered thread.

    Scheduled Pinned Locked Moved Solved
    FOG Problems
    8
    41
    16.2k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • d4rk3D
      d4rk3 @Wayne Workman
      last edited by

      @Wayne-Workman Even if the web interface isn’t used by anyone the load will always be high.

      However, out of the ~1000 clients I have approximately 900 have the new client installed.

      Maybe this issue is related to these newest revisions with regard to communication with the new client (v0.95).

      1 Reply Last reply Reply Quote 0
      • Tom ElliottT
        Tom Elliott
        last edited by

        Can all update? My suspicion as to a part of the problem is specific to the fog multicast service. I noticed things were not starting for it and would continuously be respawned. This would create quite a high load on the system as it would be constantly creating new sessions and spawning new instances. While I’m sure there may be some cases of weirdness I think this may be a largely attributing factor.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        1 Reply Last reply Reply Quote 0
        • Joseph HalesJ
          Joseph Hales Testers
          last edited by Joseph Hales

          I am still seeing the issue after updating and restarting SVN 5106. Also after updating to 5108.

          RTFM

          1 Reply Last reply Reply Quote 0
          • TrevelyanT
            Trevelyan
            last edited by

            Still seeing the issue too. Note the huge number of threads and tasks!

            still.png

            1 Reply Last reply Reply Quote 0
            • Tom ElliottT
              Tom Elliott
              last edited by

              Guys,

              Do me a favor, it’s going to break things, but it’s very specific.

              It’s the service module active checks. They’re spawning way too many processes.

              For now, just run: mv /var/www/fog/service/servicemodule-active.php /var/www/fog/service/service-active.php

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              1 Reply Last reply Reply Quote 0
              • TrevelyanT
                Trevelyan
                last edited by

                Well it broke it but it stopped the “thing”! 🙂

                Tom ElliottT 1 Reply Last reply Reply Quote 2
                • Tom ElliottT
                  Tom Elliott @Trevelyan
                  last edited by

                  @Trevelyan I’m aware of it breaking the client stuff, but it’s all i can do for right now.

                  I’m remoted into another system having the same type of issues and trying to use it as a way to figure out exactly what’s up. You can image, and do all that, just no FOG Client stuff at the moment. I’ll be trying to narrow exactly where it is.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  1 Reply Last reply Reply Quote 0
                  • Tom ElliottT
                    Tom Elliott
                    last edited by

                    While it’s not perfect, I found 4168 is still good and 4169 things are “decent”. 4170, all hell breaks loose. Please update back to 4168 to be “functional”.

                    You can do this with svn by cd /opt/trunk;svn up -r 4168;cd bin;./installfog.sh

                    Of course change the /opt/trunk path to the location of your svn folder.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    1 Reply Last reply Reply Quote 1
                    • A
                      Adam Taylor
                      last edited by

                      4168 == sanity!

                      Never get past about 30% now.

                      Adam

                      1 Reply Last reply Reply Quote 1
                      • Joseph HalesJ
                        Joseph Hales Testers
                        last edited by

                        Ok im functional at 4168 just a note all the snapins that weren’t showing up are all there now.

                        RTFM

                        1 Reply Last reply Reply Quote 1
                        • Tom ElliottT
                          Tom Elliott
                          last edited by Tom Elliott

                          I’m now fairly certain these cpu load issues is now fixed. It was due to HookManager. I’m sorry it took so long to figure out.

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          1 Reply Last reply Reply Quote 1
                          • TrevelyanT
                            Trevelyan
                            last edited by

                            Everything seems happy now!

                            Thanks 💃

                            PS - Hosts and Users report shows up as error 500 - links to (fog server…) /node=report&sub=file&f=SG9zdHMgYW5kIFVzZXJzLnBocA==

                            Tom ElliottT 1 Reply Last reply Reply Quote 1
                            • Tom ElliottT
                              Tom Elliott @Trevelyan
                              last edited by

                              @Trevelyan I’ll fix the hosts and users report when I get into work.

                              Can you all verify if things are better and let me know? I’m sorry about the problem this caused, but hey progress or something.

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              M 1 Reply Last reply Reply Quote 3
                              • S
                                Sebastian Roth Moderator
                                last edited by

                                I want to thank Tom again for this! I tried to assist and help but I just don’t know the code (and its history) as much as Tom does. He’s done all the hard work to find and fix this issue. Along the way he also fixed a couple other things as well.

                                @Tom-Elliott Don’t say sorry. FOG is work in progress and you are pushing things way forward!! :metal:

                                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                1 Reply Last reply Reply Quote 4
                                • M
                                  Mark Buteyn @Tom Elliott
                                  last edited by

                                  @Tom-Elliott Thanks for all your work on fog. Long time user. I’m on 5124 and CPU utilization looks good. I echo @Trevelyan 's concern about the user tracker report. CPU utilization goes high and then comes back down after the 500 error is sent to the browser. I know you said you’d fix it when you got to work. Is there a way for me to find out what version fixes it? Then I won’t bother posting this question in the forum.

                                  Thanks,

                                  Mark

                                  1 Reply Last reply Reply Quote 0
                                  • Joseph HalesJ
                                    Joseph Hales Testers
                                    last edited by

                                    I updated my main and storage node after the update to SVN 5126 the CPU usage is normal but something is still wrong with mysql the database hits MAX_Workers right away and this prevents the storage node from connecting.
                                    upload-4f016a73-18f4-44cf-bf2d-9be1142a2c66

                                    [Thu Oct 29 13:34:44.593182 2015] [:error] [pid 21227] [client 10.109.49.21:54036] PHP Warning:  mysqli::mysqli(): MySQL server has gone away in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:34:44.593262 2015] [:error] [pid 21227] [client 10.109.49.21:54036] PHP Warning:  mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:34:44.681319 2015] [:error] [pid 21145] [client 10.114.51.136:49375] PHP Warning:  mysqli::mysqli(): MySQL server has gone away in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:34:44.681414 2015] [:error] [pid 21145] [client 10.114.51.136:49375] PHP Warning:  mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:34:44.837520 2015] [mpm_prefork:notice] [pid 21097] AH00169: caught SIGTERM, shutting down
[Thu Oct 29 13:36:56.973677 2015] [mpm_prefork:notice] [pid 1510] AH00163: Apache/2.4.16 (Ubuntu) OpenSSL/1.0.1f configured -- resuming normal operations
[Thu Oct 29 13:36:56.981375 2015] [core:notice] [pid 1510] AH00094: Command line: '/usr/sbin/apache2'
[Thu Oct 29 13:38:00.215860 2015] [mpm_prefork:error] [pid 1510] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting```

                                    RTFM

                                    1 Reply Last reply Reply Quote 0
                                    • Joseph HalesJ
                                      Joseph Hales Testers
                                      last edited by

                                      After reverting storage node is back up.
                                      upload-7f6a2cad-8b16-4f37-be64-1c53f366f158

                                      but the apache log show slightly different errors but the node is working?

                                      [Thu Oct 29 13:58:49.245894 2015] [:error] [pid 2338] [client 10.44.32.31:61425] PHP Strict Standards:  Declaration of FOGController::getSubObjectIDs() should be compatible with FOGBase::getSubObjectIDs($object = 'Host', $findWhere = Array, $getField = 'id', $not = false, $operator = 'AND', $orderBy = 'name') in /var/www/html/fog/lib/fog/FOGController.class.php on line 0
[Thu Oct 29 13:58:49.245960 2015] [:error] [pid 2338] [client 10.44.32.31:61425] PHP Fatal error:  Access level to FOGManagerController::orderBy() must be public (as in class FOGBase) in /var/www/html/fog/lib/fog/FOGManagerController.class.php on line 2
[Thu Oct 29 13:58:49.319201 2015] [:error] [pid 2222] [client 10.102.50.209:53776] PHP Warning:  mysqli::mysqli(): MySQL server has gone away in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:58:49.320596 2015] [:error] [pid 2222] [client 10.102.50.209:53776] PHP Strict Standards:  Declaration of FOGController::getSubObjectIDs() should be compatible with FOGBase::getSubObjectIDs($object = 'Host', $findWhere = Array, $getField = 'id', $not = false, $operator = 'AND', $orderBy = 'name') in /var/www/html/fog/lib/fog/FOGController.class.php on line 0
[Thu Oct 29 13:58:49.320645 2015] [:error] [pid 2222] [client 10.102.50.209:53776] PHP Fatal error:  Access level to FOGManagerController::orderBy() must be public (as in class FOGBase) in /var/www/html/fog/lib/fog/FOGManagerController.class.php on line 2
[Thu Oct 29 13:58:49.362058 2015] [:error] [pid 2317] [client 10.41.32.7:64706] PHP Strict Standards:  Declaration of FOGController::getSubObjectIDs() should be compatible with FOGBase::getSubObjectIDs($object = 'Host', $findWhere = Array, $getField = 'id', $not = false, $operator = 'AND', $orderBy = 'name') in /var/www/html/fog/lib/fog/FOGController.class.php on line 0
[Thu Oct 29 13:58:49.362127 2015] [:error] [pid 2317] [client 10.41.32.7:64706] PHP Fatal error:  Access level to FOGManagerController::orderBy() must be public (as in class FOGBase) in /var/www/html/fog/lib/fog/FOGManagerController.class.php on line 2
[Thu Oct 29 13:58:49.402927 2015] [:error] [pid 2281] [client 10.43.32.103:64848] PHP Warning:  mysqli::mysqli(): MySQL server has gone away in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Thu Oct 29 13:58:49.404219 2015] [:error] [pid 2281] [client 10.43.32.103:64848] PHP Strict Standards:  Declaration of FOGController::getSubObjectIDs() should be compatible with FOGBase::getSubObjectIDs($object = 'Host', $findWhere = Array, $getField = 'id', $not = false, $operator = 'AND', $orderBy = 'name') in /var/www/html/fog/lib/fog/FOGController.class.php on line 0
[Thu Oct 29 13:58:49.404267 2015] [:error] [pid 2281] [client 10.43.32.103:64848] PHP Fatal error:  Access level to FOGManagerController::orderBy() must be public (as in class FOGBase) in /var/www/html/fog/lib/fog/FOGManagerController.class.php on line 2
[Thu Oct 29 13:58:49.415663 2015] [:error] [pid 2223] [client 10.1.50.216:51730] PHP Strict Standards:  Declaration of FOGController::getSubObjectIDs() should be compatible with FOGBase::getSubObjectIDs($object = 'Host', $findWhere = Array, $getField = 'id', $not = false, $operator = 'AND', $orderBy = 'name') in /var/www/html/fog/lib/fog/FOGController.class.php on line 0
[Thu Oct 29 13:58:49.415714 2015] [:error] [pid 2223] [client 10.1.50.216:51730] PHP Fatal error:  Access level to FOGManagerController::orderBy() must be public (as in class FOGBase) in /var/www/html/fog/lib/fog/FOGManagerController.class.php on line 2
[Thu Oct 29 13:58:49.897820 2015] [mpm_prefork:notice] [pid 1510] AH00169: caught SIGTERM, shutting down
[Thu Oct 29 13:58:50.626613 2015] [mpm_prefork:notice] [pid 4285] AH00163: Apache/2.4.16 (Ubuntu) OpenSSL/1.0.1f configured -- resuming normal operations
[Thu Oct 29 13:58:50.626657 2015] [core:notice] [pid 4285] AH00094: Command line: '/usr/sbin/apache2'
[Thu Oct 29 13:59:00.773873 2015] [mpm_prefork:error] [pid 4285] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting```

                                      RTFM

                                      Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                                      • Wayne WorkmanW
                                        Wayne Workman @Joseph Hales
                                        last edited by

                                        @Joseph-Hales Does your max_connections settings get wiped when you update? Can you check them?

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                        Daily Clean Installation Results:
                                        https://fogtesting.fogproject.us/
                                        FOG Reporting:
                                        https://fog-external-reporting-results.fogproject.us/

                                        1 Reply Last reply Reply Quote 0
                                        • Joseph HalesJ
                                          Joseph Hales Testers
                                          last edited by

                                          I checked max_workers and max_connections were still edited properly to test further I set max_workers and max_connections to 1000 and I am still seeing the same issue.

                                          RTFM

                                          1 Reply Last reply Reply Quote 0
                                          • Joseph HalesJ
                                            Joseph Hales Testers
                                            last edited by

                                            SVN 5155 CPU is fine but storage node still can’t connect and max_workers hits that limit again even when set to 1000.

                                            

[Fri Oct 30 14:02:09.746764 2015] [:error] [pid 11232] [client 10.1.50.108:63311] PHP Warning:  mysqli::mysqli(): MySQL server has gone away in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Fri Oct 30 14:02:09.746866 2015] [:error] [pid 11232] [client 10.1.50.108:63311] PHP Warning:  mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Fri Oct 30 14:02:09.784735 2015] [:error] [pid 11087] [client 10.109.49.5:59983] PHP Warning:  mysqli::mysqli(): MySQL server has gone away in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Fri Oct 30 14:02:09.784862 2015] [:error] [pid 11087] [client 10.109.49.5:59983] PHP Warning:  mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Fri Oct 30 14:02:09.839016 2015] [:error] [pid 14086] [client 10.43.48.199:63489] PHP Warning:  mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
[Fri Oct 30 14:02:10.027273 2015] [mpm_prefork:notice] [pid 11050] AH00169: caught SIGTERM, shutting down
[Fri Oct 30 14:04:21.875728 2015] [mpm_prefork:notice] [pid 1544] AH00163: Apache/2.4.16 (Ubuntu) OpenSSL/1.0.1f configured -- resuming normal operations
[Fri Oct 30 14:04:21.888867 2015] [core:notice] [pid 1544] AH00094: Command line: '/usr/sbin/apache2'
[Fri Oct 30 14:05:08.065535 2015] [mpm_prefork:error] [pid 1544] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting```

                                            RTFM

                                            1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 1 / 3
                                            • First post
                                              Last post

                                            207

                                            Online

                                            11.7k

                                            Users

                                            17.1k

                                            Topics

                                            154.6k

                                            Posts
                                            Copyright © 2012-2024 FOG Project