• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

High CPU usage from multiple httpd processes version 5800

Scheduled Pinned Locked Moved Solved
FOG Problems
5
22
6.2k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • T
    Tom Elliott
    last edited by Mar 11, 2016, 1:31 PM

    Please restart the FOGScheduler service. Or stop it. It should drop the httpd processes for you. I’m still trying to figure out what’s causing IT to be the culprit here.

    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

    K 2 Replies Last reply Mar 11, 2016, 1:40 PM Reply Quote 0
    • K
      Kiweegie @Tom Elliott
      last edited by Mar 11, 2016, 1:40 PM

      @Tom-Elliott HI Tom - I’ve restarted the server entirely twice which didn’t help. I’ve stopped the FOGScheduler service now and watching top to see if the httpd processes disappear but so far not happening.

      Will post back in 10 with further report of above.

      cheers Kiweegie.

      1 Reply Last reply Reply Quote 0
      • K
        Kiweegie @Tom Elliott
        last edited by Mar 11, 2016, 2:44 PM

        @Tom-Elliott Sorry got sidetracked by lunch 🙂

        Still several (40+) httpd processes running. FOGScheduler service is off.

        Checking /var/log/httpd/access_log shows the following connections from multiple clients

        10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/snapins.checkin.php?mac=D4:3D:7E:25:49:7F%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 4 "-" "-"
        10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/Printers.php?mac=D4:3D:7E:38:FE:60%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 12 "-" "-"
        10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/greenfog.php?mac=74:27:EA:AD:AC:0A%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 73 "-" "-"
        10.***.***.***- - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/autologout.php?mac=A4:1F:72:86:3A:A4%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
        10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/autologout.php?mac=74:27:EA:CF:9A:FA%7C%7C00:00:00:00:00:00:00:E0%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
        10.***.***.*** - - [11/Mar/2016:14:26:50 +0000] "GET /fog/service/servicemodule-active.php?moduleid=taskreboot&mac=A4:1F:72:86:5C:CB%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 5 "-" "-"
        10.***.***.*** - - [11/Mar/2016:14:26:51 +0000] "GET /fog/service/autologout.php?mac=A4:1F:72:86:5C:CB%7C%7C00:00:00:00:00:00:00:E0&newService=1 HTTP/1.1" 200 6 "-" "-"
        10.***.***.*** - - [11/Mar/2016:14:26:53 +0000] "GET /fog/status/bandwidth.php?dev=eth0 HTTP/1.1" 200 47 "-" "-"
        10.***.***.*** - - [11/Mar/2016:14:26:51 +0000] "GET /fog/service/printerlisting.php?newService=1 HTTP/1.1" 200 5 "-" "-"
        

        And in /var/log/httpd/error_log seeing this

        [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::real_connect(): (08004/1040): Too many connections in /var/www/html/fog/lib/db/mysql.class.php on line 30
        [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::set_charset(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 31
        [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
        [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
        [Fri Mar 11 13:11:46 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 52
        [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::real_connect(): (08004/1040): Too many connections in /var/www/html/fog/lib/db/mysql.class.php on line 30
        [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::set_charset(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 31
        [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
        [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::select_db(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 40
        [Fri Mar 11 13:11:48 2016] [error] [client 10.***.***.***] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 52
        df: `/home/fog/images': Permission denied
        df: no file systems processed
        [Fri Mar 11 13:22:15 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
        [Fri Mar 11 13:22:17 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
        [Fri Mar 11 13:22:19 2016] [warn] child process 2991 still did not exit, sending a SIGTERM
        [Fri Mar 11 13:22:21 2016] [error] child process 2991 still did not exit, sending a SIGKILL
        [Fri Mar 11 13:22:22 2016] [notice] caught SIGTERM, shutting down
        [Fri Mar 11 13:23:30 2016] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
        [Fri Mar 11 13:23:30 2016] [notice] Digest: generating secret for digest authentication ...
        [Fri Mar 11 13:23:30 2016] [notice] Digest: done
        [Fri Mar 11 13:23:31 2016] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.6.19 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
        df: `/home/fog/images': Permission denied
        df: no file systems processed
        

        From another post I’ve looked at max connections in mysql (this is a CentOS 6.7 server)

        su mysql
        ulimit -n
        1024
        
        ulimit -a
        core file size          (blocks, -c) 0
        data seg size           (kbytes, -d) unlimited
        scheduling priority             (-e) 0
        file size               (blocks, -f) unlimited
        pending signals                 (-i) 7414
        max locked memory       (kbytes, -l) 64
        max memory size         (kbytes, -m) unlimited
        open files                      (-n) 1024
        pipe size            (512 bytes, -p) 8
        POSIX message queues     (bytes, -q) 819200
        real-time priority              (-r) 0
        stack size              (kbytes, -s) 10240
        cpu time               (seconds, -t) unlimited
        max user processes              (-u) 1024
        virtual memory          (kbytes, -v) unlimited
        file locks                      (-x) unlimited
        

        Does this look ok? Should I look to increase the max connections above 1024?

        regards Kiweegie.

        1 Reply Last reply Reply Quote 0
        • K
          Kiweegie
          last edited by Mar 11, 2016, 3:27 PM

          Possible red-herring but worthwhile raising the question I think.

          watching connections via

          ss -tp
          

          throws up amongst a host of other connections

          CLOSE-WAIT 1      0           Fog-server-IP:45260        54.93.153.156:http
          CLOSE-WAIT 1      0           Fog-server-IP:45856        54.93.153.156:http
          
          C:\Windows\System32>nslookup 54.93.153.156
          Server:  my_AD_Domain controller
          Address:   my_AD_Domain controller_IP
          
          Name:    ec2-54-93-153-156.eu-central-1.compute.amazonaws.com
          Address:  54.93.153.156
          

          for what reason would FOG be looking to connect to an Amazon IP??? I’m seeing a similar connection attempt to 52.28.249.93 which is another Amazon IP.

          There are a load of ESTABLISHED connections showing like below which I believe to be the actual httpd processes though.

          Proto Recv-Q Send-Q Local Address                Foreign Address             State
          tcp        0      0 ::ffff:Fof_Server_IP:80     ::ffff:Client-IP:64981 ESTABLISHED
          

          If you need any other logs etc to help get to the bottom of this I’m more than happy to oblige.

          cheers Kiweegie.

          T 1 Reply Last reply Mar 11, 2016, 3:43 PM Reply Quote 0
          • T
            Tom Elliott @Kiweegie
            last edited by Mar 11, 2016, 3:43 PM

            @Kiweegie is your fog server reachable from the Internet?

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            K 1 Reply Last reply Mar 11, 2016, 3:47 PM Reply Quote 0
            • K
              Kiweegie @Tom Elliott
              last edited by Mar 11, 2016, 3:47 PM

              @Tom-Elliott Nope, not without vpn to our corporate network

              T W 2 Replies Last reply Mar 11, 2016, 3:48 PM Reply Quote 0
              • T
                Tom Elliott @Kiweegie
                last edited by Mar 11, 2016, 3:48 PM

                @Kiweegie There is a couple things the reach out to the internet, one is the FOG Services (Scheduler, MulticastManager, etc…) that reach out to the internet to get the system’s external address in the case that’s the way your node is setup to be reached, so I doubt those are the “problem” things.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                K 1 Reply Last reply Mar 14, 2016, 10:18 AM Reply Quote 0
                • W
                  Wayne Workman @Kiweegie
                  last edited by Wayne Workman Mar 11, 2016, 5:15 PM Mar 11, 2016, 11:14 PM

                  @Kiweegie Turn down your client checkin time. At work, I’ve got mine set to 300 seconds. IMHO, 60 seconds is absurd.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                  Daily Clean Installation Results:
                  https://fogtesting.fogproject.us/
                  FOG Reporting:
                  https://fog-external-reporting-results.fogproject.us/

                  G K 2 Replies Last reply Mar 11, 2016, 11:51 PM Reply Quote 0
                  • G
                    george1421 Moderator @Wayne Workman
                    last edited by Mar 11, 2016, 11:51 PM

                    FWIW, I have mine set to 900 seconds. I don’t deploy snapins post deployment or issue unattended deployments. There is no real value (in my environment) to have a client check-ins faster than every 15 minutes.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                    1 Reply Last reply Reply Quote 0
                    • K
                      Kiweegie @Tom Elliott
                      last edited by Mar 14, 2016, 10:18 AM

                      @Tom-Elliott Morning Tom not sure if related to changes you made for me late last week or a new issue but since the teamviewer session we had, deployment of images fails with error below:

                      failed to create deployment tasking....

                      The offending line in path /var/www/html/fog/lib/fog/fogftpclass.php is on line 104

                            if (@ftp_login(self::$link,$username,$password) === false) self::ftperror();
                      

                      no credentials have changed at all. Snapins deploy fine, only images are affected and we have tried with multiple images.

                      regards, Kiweegie.

                      T 1 Reply Last reply Mar 14, 2016, 10:50 AM Reply Quote 0
                      • T
                        Tom Elliott @Kiweegie
                        last edited by Mar 14, 2016, 10:50 AM

                        @Kiweegie I’m not able to replicate this. If snapins are working off of the same nodes that are not sending the image, then is something potentially wrong with how the images are mounted on that system? I know it’s stating an FTP error, so it’s a wee bit odd.

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        K 1 Reply Last reply Mar 14, 2016, 1:53 PM Reply Quote 0
                        • K
                          Kiweegie @Wayne Workman
                          last edited by Mar 14, 2016, 11:22 AM

                          @Wayne-Workman @george1421 Hi Wayne/George

                          Where do I find the client checkin time setting? I’ve checked fog settings and nothing shows up there. Could you point me in the right direction please?

                          cheers Kiweegie

                          T G 2 Replies Last reply Mar 14, 2016, 11:24 AM Reply Quote 0
                          • T
                            Tom Elliott @Kiweegie
                            last edited by Mar 14, 2016, 11:24 AM

                            @Kiweegie FOG Configuration Page->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME,FOG_GRACE_TIMEOUT

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                            1 Reply Last reply Reply Quote 1
                            • G
                              george1421 Moderator @Kiweegie
                              last edited by Mar 14, 2016, 11:26 AM

                              @Kiweegie FOG Configuration->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                              K 1 Reply Last reply Mar 14, 2016, 1:23 PM Reply Quote 0
                              • K
                                Kiweegie @george1421
                                last edited by Mar 14, 2016, 1:23 PM

                                @george1421 said:

                                FOG Configuration->FOG Settings->FOG Service->FOG_SERVICE_CHECKIN_TIME

                                cheers George!

                                1 Reply Last reply Reply Quote 0
                                • K
                                  Kiweegie @Tom Elliott
                                  last edited by Mar 14, 2016, 1:53 PM

                                  @Tom-Elliott Hi Tom unfortunately I was going off the information passed to me by my service desk… which turns out to not be as accurate as I thought 😞

                                  I’ve tried deploying snapin and while there is no error thrown up in the GUI, c:\fog.log on target machine shows the same error as seen when trying to deploy image.

                                   14/03/2016 13:35 Middleware::Communication Unknown Response: Type: 2, File: /var/www/html/fog/lib/fog/fogftp.class.php, Line: 104, Message: ftp_login(): Login incorrect.
                                  

                                  This would suggest that the fog or ftp user password has been changed but it has not been altered at least not by me. Entry in following 2 locations matched what I had in keepass

                                  Storage Management > All Storage nodes > DefaultMember > Management Password
                                  FOG Configuration > FOG Settings > TFTP Server > FOG_TFTP_FTP_Password

                                  However, resetting the fog user password at commandline fixed the issue…

                                  Don’t understand who or what amended that password as only I have access to the box and it sure wasn’t me!

                                  cheers . Kiweegie.

                                  W 1 Reply Last reply Mar 14, 2016, 5:11 PM Reply Quote 0
                                  • W
                                    Wayne Workman @Kiweegie
                                    last edited by Mar 14, 2016, 5:11 PM

                                    @Kiweegie The installer a week to two weeks ago set it wrong if there were certain special characters present in the password. It’s since been fixed though. You should verify that what is inside of .fogsettings is correct, and what shows up for the user/pass in the storage management area is correct, and for the kernel update stuff too. All of this is outlined in the “Troubleshoot FTP” wiki article.

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                    Daily Clean Installation Results:
                                    https://fogtesting.fogproject.us/
                                    FOG Reporting:
                                    https://fog-external-reporting-results.fogproject.us/

                                    K 1 Reply Last reply Mar 14, 2016, 5:33 PM Reply Quote 0
                                    • S
                                      Sebastian Roth Moderator
                                      last edited by Sebastian Roth Mar 14, 2016, 11:14 AM Mar 14, 2016, 5:12 PM

                                      @Kiweegie I vaguely remember @Wayne-Workman having seen a fog user password change after running the installer script at some point. Maybe this is related? https://forums.fogproject.org/topic/6649/storage-node-installation-resets-local-fog-user-s-password-to-something-wrong

                                      Cannot actually find the commit that was made to fix this but from what Tom says I thought this was fixed somewhere around version 63xx. Maybe not? Please keep an eye on this and let us know!

                                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                      1 Reply Last reply Reply Quote 1
                                      • K
                                        Kiweegie @Wayne Workman
                                        last edited by Mar 14, 2016, 5:33 PM

                                        @Wayne-Workman HI Wayne thanks for the heads up. fog password in this case is purely alphanumeric though so don’t think that was it. Resetting the fog password has resolved my issue however so happy to have this one marked as solved.

                                        regards Kiweegie.

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          Sebastian Roth Moderator
                                          last edited by Mar 14, 2016, 5:36 PM

                                          @Wayne-Workman Should have reloaded the page… 😉 Good to know that you seam to have verified that this is fixed. Probably it was still wrong in .fogsettings for @Kiweegie?!

                                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                          T 1 Reply Last reply Mar 14, 2016, 5:40 PM Reply Quote 0
                                          • 1
                                          • 2
                                          • 2 / 2
                                          2 / 2
                                          • First post
                                            11/22
                                            Last post

                                          161

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project