• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

High MySQL CPU Usage Bogging Down Server

Scheduled Pinned Locked Moved Solved
FOG Problems
6
47
4.2k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • U
    UWPVIOLATOR @Tom Elliott
    last edited by Feb 19, 2018, 9:04 PM

    @tom-elliott No our admin just upped that on Friday to.

    0_1519074215579_1a99c380-c965-4e13-8add-390a722345d2-image.png

    W 1 Reply Last reply Feb 19, 2018, 9:23 PM Reply Quote 0
    • W
      Wayne Workman @UWPVIOLATOR
      last edited by Feb 19, 2018, 9:23 PM

      @uwpviolator This would retrieve logs for MySQL:
      journalctl -u mysql.service Or if using mariadb: journalctl -u mariadb.service
      Anything interesting in there? Use your arrow keys to navigate up/down/left/right.

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
      Daily Clean Installation Results:
      https://fogtesting.fogproject.us/
      FOG Reporting:
      https://fog-external-reporting-results.fogproject.us/

      U 1 Reply Last reply Feb 19, 2018, 9:25 PM Reply Quote 0
      • U
        UWPVIOLATOR @Wayne Workman
        last edited by Feb 19, 2018, 9:25 PM

        @wayne-workman said in High MySQL CPU Usage Bogging Down Server:

        journalctl -u mariadb.service

        Nope it just says start and stop service.

        -- Logs begin at Mon 2018-02-19 12:13:31 CST, end at Mon 2018-02-19 15:24:26 CST. --
Feb 19 12:13:33 FogDB systemd[1]: Starting MySQL Community Server...
Feb 19 12:13:37 FogDB systemd[1]: Started MySQL Community Server.
Feb 19 13:13:01 FogDB systemd[1]: Stopping MySQL Community Server...
Feb 19 13:13:08 FogDB systemd[1]: Stopped MySQL Community Server.
Feb 19 13:13:08 FogDB systemd[1]: Starting MySQL Community Server...
Feb 19 13:13:10 FogDB systemd[1]: Started MySQL Community Server.
        1 Reply Last reply Reply Quote 0
        • U
          UWPVIOLATOR
          last edited by Feb 19, 2018, 9:27 PM

          @Tom-Elliott @Wayne-Workman I have turned back on the FOGService so come tomorrow morning when the devices boot again they should all try to connect like normal. Will check in and see what it looks like in the am. Maybe it was that all the encryptions broke and that was all it needed after getting the server to settle back down so they could authenticate.

          1 Reply Last reply Reply Quote 0
          • U
            UWPVIOLATOR
            last edited by Feb 20, 2018, 12:38 AM

            @Tom-Elliott @Wayne-Workman Here is the end of the day Apache Log. Anything pop out to you there?

            [Mon Feb 19 13:13:08.618425 2018] [php7:warn] [pid 4738] [client 10.106.151.97:51232] PHP Warning:  PDOStatement::execute(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 582
[Mon Feb 19 13:13:08.618477 2018] [php7:warn] [pid 4738] [client 10.106.151.97:51232] PHP Warning:  PDOStatement::execute(): Error reading result set's header in /var/www/html/fog/lib/db/pdodb.class.php on line 582
[Mon Feb 19 13:13:08.642636 2018] [php7:warn] [pid 4565] [client 10.75.1.49:51793] PHP Warning:  PDOStatement::execute(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 582
[Mon Feb 19 13:13:08.642677 2018] [php7:warn] [pid 4565] [client 10.75.1.49:51793] PHP Warning:  PDOStatement::execute(): Error reading result set's header in /var/www/html/fog/lib/db/pdodb.class.php on line 582
[Mon Feb 19 13:13:08.684912 2018] [php7:warn] [pid 4299] [client 10.116.151.206:56595] PHP Warning:  PDOStatement::execute(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 582
[Mon Feb 19 13:13:08.684974 2018] [php7:warn] [pid 4299] [client 10.116.151.206:56595] PHP Warning:  PDOStatement::execute(): Error reading result set's header in /var/www/html/fog/lib/db/pdodb.class.php on line 582
[Mon Feb 19 13:13:09.096405 2018] [php7:warn] [pid 3858] [client 10.105.150.66:55526] PHP Warning:  PDOStatement::execute(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 582
[Mon Feb 19 13:13:09.096461 2018] [php7:warn] [pid 3858] [client 10.105.150.66:55526] PHP Warning:  PDOStatement::execute(): Error reading result set's header in /var/www/html/fog/lib/db/pdodb.class.php on line 582
[Mon Feb 19 13:13:09.299891 2018] [php7:error] [pid 1236] [client 10.107.1.19:44883] PHP Fatal error:  Uncaught Error: Call to a member function lastInsertId() on boolean in /var/www/html/fog/lib/db/pdodb.class.php:443\nStack trace:\n#0 /var/www/html/fog/lib/fog/fogcontroller.class.php(527): PDODB->insertId()\n#1 /var/www/html/fog/lib/fog/hookmanager.class.php(59): FOGController->save()\n#2 /var/www/html/fog/lib/fog/bootmenu.class.php(1157): HookManager->processEvent('IPXE_EDIT', Array)\n#3 /var/www/html/fog/lib/fog/bootmenu.class.php(197): BootMenu->_parseMe(Array)\n#4 /var/www/html/fog/service/ipxe/boot.php(35): BootMenu->__construct(Object(Host))\n#5 {main}\n  thrown in /var/www/html/fog/lib/db/pdodb.class.php on line 443
[Mon Feb 19 13:13:09.883195 2018] [php7:error] [pid 3886] [client 10.88.1.67:52314] PHP Fatal error:  Uncaught Error: Call to a member function lastInsertId() on boolean in /var/www/html/fog/lib/db/pdodb.class.php:443\nStack trace:\n#0 /var/www/html/fog/lib/fog/fogcontroller.class.php(527): PDODB->insertId()\n#1 /var/www/html/fog/lib/fog/hookmanager.class.php(59): FOGController->save()\n#2 /var/www/html/fog/lib/fog/bootmenu.class.php(1157): HookManager->processEvent('IPXE_EDIT', Array)\n#3 /var/www/html/fog/lib/fog/bootmenu.class.php(197): BootMenu->_parseMe(Array)\n#4 /var/www/html/fog/service/ipxe/boot.php(35): BootMenu->__construct(Object(Host))\n#5 {main}\n  thrown in /var/www/html/fog/lib/db/pdodb.class.php on line 443
[Mon Feb 19 13:13:10.104024 2018] [php7:error] [pid 3977] [client 10.84.1.2:54487] PHP Fatal error:  Uncaught Error: Call to a member function lastInsertId() on boolean in /var/www/html/fog/lib/db/pdodb.class.php:443\nStack trace:\n#0 /var/www/html/fog/lib/fog/fogcontroller.class.php(527): PDODB->insertId()\n#1 /var/www/html/fog/lib/fog/hookmanager.class.php(59): FOGController->save()\n#2 /var/www/html/fog/lib/fog/bootmenu.class.php(1157): HookManager->processEvent('IPXE_EDIT', Array)\n#3 /var/www/html/fog/lib/fog/bootmenu.class.php(197): BootMenu->_parseMe(Array)\n#4 /var/www/html/fog/service/ipxe/boot.php(35): BootMenu->__construct(Object(Host))\n#5 {main}\n  thrown in /var/www/html/fog/lib/db/pdodb.class.php on line 443
[Mon Feb 19 13:14:25.954217 2018] [core:warn] [pid 982] AH00045: child process 3981 still did not exit, sending a SIGTERM
[Mon Feb 19 13:14:27.961702 2018] [core:warn] [pid 982] AH00045: child process 3981 still did not exit, sending a SIGTERM
[Mon Feb 19 13:14:29.963817 2018] [core:warn] [pid 982] AH00045: child process 3981 still did not exit, sending a SIGTERM
[Mon Feb 19 13:14:31.965940 2018] [core:error] [pid 982] AH00046: child process 3981 still did not exit, sending a SIGKILL
[Mon Feb 19 13:14:32.970135 2018] [mpm_prefork:notice] [pid 982] AH00169: caught SIGTERM, shutting down
[Mon Feb 19 13:14:33.576263 2018] [mpm_prefork:notice] [pid 15054] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.0g configured -- resuming normal operations
[Mon Feb 19 13:14:33.576327 2018] [core:notice] [pid 15054] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 19 13:15:44.013030 2018] [mpm_prefork:error] [pid 15054] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Mon Feb 19 16:48:45.938252 2018] [php7:error] [pid 16421] [client 10.138.150.19:50439] PHP Fatal error:  Uncaught Exception: #!im in /var/www/html/fog/lib/fog/fogbase.class.php:519\nStack trace:\n#0 /var/www/html/fog/lib/client/registerclient.class.php(46): FOGBase::getHostItem(true, false, false, true)\n#1 /var/www/html/fog/lib/fog/fogpage.class.php(2843): RegisterClient->json()\n#2 /var/www/html/fog/lib/fog/fogpage.class.php(267): FOGPage->requestClientInfo()\n#3 /var/www/html/fog/lib/pages/processlogin.class.php(57): FOGPage->__construct('')\n#4 [internal function]: ProcessLogin->__construct()\n#5 /var/www/html/fog/lib/fog/fogbase.class.php(457): ReflectionClass->newInstanceArgs(Array)\n#6 /var/www/html/fog/management/index.php(31): FOGBase::getClass('ProcessLogin')\n#7 {main}\n  thrown in /var/www/html/fog/lib/fog/fogbase.class.php on line 519
            T 1 Reply Last reply Feb 20, 2018, 1:12 AM Reply Quote 0
            • T
              Tom Elliott @UWPVIOLATOR
              last edited by Tom Elliott Feb 19, 2018, 11:16 PM Feb 20, 2018, 1:12 AM

              @uwpviolator The very last line is interesting as it’s bascially saying the mac being passed is invalid (#!im = invalid mac) though this should be handled if it’s a new client getting this message, that’s not to say there isn’t a problem, just trying to understand why it’s saying that in the first place. But I highly doubt it’s that that’s causing the problem with high cpu as it’s the only message. The more interesting is the MaxRequestWorkers. Typically I see this when a bunch of clients are trying to checkin at the same time, and my guess is this is kind of the case. They’re stuck in an authentication loop, which hopefully resetting encryption data might help fix. (Initially it won’t help as all clients would still be trying to checkin to get their information, but once it stabilizes you should see a significant difference).
              What’s interesting to me is the client checkin time.

              Our defaults are set to reauthenticate data every 30 minutes. And your check-in cycle time is set to 20 minutes, from what I can gather. Maybe it’s the time cycles are “too much in sync” with one another? I’m just speculating at this point. The time itself shouldn’t be a major problem, but if you have a bunch of hosts authenticating all at the same time I could see this being a potential bottleneck.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              W 1 Reply Last reply Feb 20, 2018, 4:16 AM Reply Quote 0
              • W
                Wayne Workman @Tom Elliott
                last edited by Wayne Workman Feb 19, 2018, 10:16 PM Feb 20, 2018, 4:16 AM

                @tom-elliott said in High MySQL CPU Usage Bogging Down Server:

                Our defaults are set to reset encryption data every 30 minutes

                Since when was that a thing? First I’m hearing of it.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                Daily Clean Installation Results:
                https://fogtesting.fogproject.us/
                FOG Reporting:
                https://fog-external-reporting-results.fogproject.us/

                T 1 Reply Last reply Feb 20, 2018, 4:39 AM Reply Quote 0
                • T
                  Tom Elliott @Wayne Workman
                  last edited by Tom Elliott Feb 19, 2018, 11:17 PM Feb 20, 2018, 4:39 AM

                  @wayne-workman This has always been the case. The reset is not the same as what you see done with reset encryption data on the GUI however. This is more a way to ensure a token shared is not potentially stolen and used maliciously. Updated prior post, the “reset encryption” is more properly phrased as reauthenticate every 30 minutes.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  U 1 Reply Last reply Feb 20, 2018, 2:51 PM Reply Quote 1
                  • U
                    UWPVIOLATOR @Tom Elliott
                    last edited by Feb 20, 2018, 2:51 PM

                    @tom-elliott @Wayne-Workman so far so good. Set check in time back down to 15min.

                    1 Reply Last reply Reply Quote 0
                    • S
                      Sebastian Roth Moderator
                      last edited by Mar 2, 2018, 12:33 PM

                      @UWPVIOLATOR Can we mark this solved?

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      U 1 Reply Last reply Mar 5, 2018, 5:49 PM Reply Quote 0
                      • U
                        UWPVIOLATOR @Sebastian Roth
                        last edited by Mar 5, 2018, 5:49 PM

                        @sebastian-roth I guess, we keep seeing it spike but we are going to switch from Ubuntu to Centros and then upgrade FOG to 1.5.0 in the next few weeks.

                        1 Reply Last reply Reply Quote 1
                        • U
                          UWPVIOLATOR
                          last edited by Mar 7, 2018, 6:22 PM

                          @Sebastian-Roth

                          Ok so I have more info on this. Today around 12:05-12:10 our clients lost encryption and now they are crashing the server until they check in again.

                          See this log file. I have seen this same thing on multiple machines around the same time.

                          ------------------------------------------------------------------------------
----------------------------------UserTracker---------------------------------
------------------------------------------------------------------------------
 3/7/2018 11:47 AM Client-Info Client Version: 0.11.12
 3/7/2018 11:47 AM Client-Info Client OS:      Windows
 3/7/2018 11:47 AM Client-Info Server Version: 1.4.4
 3/7/2018 11:47 AM Middleware::Response Success
------------------------------------------------------------------------------

 3/7/2018 11:47 AM Service Sleeping for 1161 seconds
 3/7/2018 12:06 PM Middleware::Communication URL: http://fogserver/fog/management/index.php?sub=requestClientInfo&configure&newService&json
 3/7/2018 12:06 PM Middleware::Communication ERROR: Could not contact FOG server
 3/7/2018 12:06 PM Middleware::Communication ERROR: Unable to connect to the remote server
 3/7/2018 12:06 PM Middleware::Response Success
 3/7/2018 12:06 PM Service ERROR: Invalid promptTime, using default
 3/7/2018 12:06 PM Middleware::Communication URL: http://fogserver/fog/management/index.php?sub=requestClientInfo&mac=10:60:4B:85:D2:35&newService&json
 3/7/2018 12:07 PM Middleware::Communication ERROR: Could not contact FOG server
 3/7/2018 12:07 PM Middleware::Communication ERROR: Unable to connect to the remote server
 3/7/2018 12:07 PM Middleware::Response Success
 3/7/2018 12:07 PM Middleware::Communication URL: http://fogserver/fog/service/getversion.php?clientver&newService&json
 3/7/2018 12:07 PM Service ERROR: Unable to get cycle data
 3/7/2018 12:07 PM Service ERROR: Unable to connect to the remote server
 3/7/2018 12:07 PM Middleware::Response Success

------------------------------------------------------------------------------
---------------------------------ClientUpdater--------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:07 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:07 PM Client-Info Client OS:      Windows
 3/7/2018 12:07 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:07 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:07 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
----------------------------------TaskReboot----------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:07 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:07 PM Client-Info Client OS:      Windows
 3/7/2018 12:07 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:07 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:07 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
--------------------------------HostnameChanger-------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:07 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:07 PM Client-Info Client OS:      Windows
 3/7/2018 12:07 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:07 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:07 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
---------------------------------SnapinClient---------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:07 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:07 PM Client-Info Client OS:      Windows
 3/7/2018 12:07 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:07 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:07 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
--------------------------------PrinterManager--------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:07 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:07 PM Client-Info Client OS:      Windows
 3/7/2018 12:07 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:07 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:07 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
--------------------------------PowerManagement-------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:07 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:07 PM Client-Info Client OS:      Windows
 3/7/2018 12:07 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:07 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:07 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
----------------------------------UserTracker---------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:07 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:07 PM Client-Info Client OS:      Windows
 3/7/2018 12:07 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:07 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:07 PM Middleware::Response ERROR: Object reference not set to an instance of an object.
 3/7/2018 12:07 PM Service Sleeping for 60 seconds
 3/7/2018 12:08 PM Middleware::Communication URL: http://fogserver/fog/management/index.php?sub=requestClientInfo&configure&newService&json
 3/7/2018 12:08 PM Middleware::Communication ERROR: Could not contact FOG server
 3/7/2018 12:08 PM Middleware::Communication ERROR: Unable to connect to the remote server
 3/7/2018 12:08 PM Middleware::Response Success
 3/7/2018 12:08 PM Service ERROR: Invalid promptTime, using default
 3/7/2018 12:08 PM Middleware::Communication URL: http://fogserver/fog/management/index.php?sub=requestClientInfo&mac=10:60:4B:85:D2:35&newService&json
 3/7/2018 12:09 PM Middleware::Communication ERROR: Could not contact FOG server
 3/7/2018 12:09 PM Middleware::Communication ERROR: Unable to connect to the remote server
 3/7/2018 12:09 PM Middleware::Response Success
 3/7/2018 12:09 PM Middleware::Communication URL: http://fogserver/fog/service/getversion.php?clientver&newService&json
 3/7/2018 12:09 PM Middleware::Communication URL: http://fogserver/fog/service/getversion.php?newService&json

 3/7/2018 12:09 PM Service Creating user agent cache
 3/7/2018 12:09 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:09 PM Middleware::Response ERROR: Object reference not set to an instance of an object.
 3/7/2018 12:09 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:09 PM Middleware::Response ERROR: Object reference not set to an instance of an object.
 3/7/2018 12:09 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:09 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
---------------------------------ClientUpdater--------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:09 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:09 PM Client-Info Client OS:      Windows
 3/7/2018 12:09 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:09 PM Middleware::Response Success
------------------------------------------------------------------------------


------------------------------------------------------------------------------
----------------------------------TaskReboot----------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:09 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:09 PM Client-Info Client OS:      Windows
 3/7/2018 12:09 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:09 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:09 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
--------------------------------HostnameChanger-------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:09 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:09 PM Client-Info Client OS:      Windows
 3/7/2018 12:09 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:09 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:09 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
---------------------------------SnapinClient---------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:09 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:09 PM Client-Info Client OS:      Windows
 3/7/2018 12:09 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:09 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:09 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
--------------------------------PrinterManager--------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:09 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:09 PM Client-Info Client OS:      Windows
 3/7/2018 12:09 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:09 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:09 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
--------------------------------PowerManagement-------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:09 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:09 PM Client-Info Client OS:      Windows
 3/7/2018 12:09 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:09 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:09 PM Middleware::Response ERROR: Object reference not set to an instance of an object.

------------------------------------------------------------------------------
----------------------------------UserTracker---------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:09 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:09 PM Client-Info Client OS:      Windows
 3/7/2018 12:09 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:09 PM Middleware::Response ERROR: Unable to get subsection
 3/7/2018 12:09 PM Middleware::Response ERROR: Object reference not set to an instance of an object.
 3/7/2018 12:09 PM Service Sleeping for 60 seconds
 3/7/2018 12:10 PM Middleware::Communication URL: http://fogserver/fog/management/index.php?sub=requestClientInfo&configure&newService&json
 3/7/2018 12:11 PM Middleware::Response Success
 3/7/2018 12:11 PM Middleware::Communication URL: http://fogserver/fog/management/index.php?sub=requestClientInfo&mac=10:60:4B:85:D2:35&newService&json
 3/7/2018 12:11 PM Middleware::Response Success
 3/7/2018 12:11 PM Middleware::Communication URL: http://fogserver/fog/service/getversion.php?clientver&newService&json
 3/7/2018 12:12 PM Middleware::Communication URL: http://fogserver/fog/service/getversion.php?newService&json

 3/7/2018 12:12 PM Service Creating user agent cache
 3/7/2018 12:12 PM Middleware::Response Module is disabled globally on the FOG server
 3/7/2018 12:12 PM Middleware::Response Module is disabled on the host
 3/7/2018 12:12 PM Middleware::Response Module is disabled globally on the FOG server

------------------------------------------------------------------------------
---------------------------------ClientUpdater--------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:12 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:12 PM Client-Info Client OS:      Windows
 3/7/2018 12:12 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:12 PM Middleware::Response Success
------------------------------------------------------------------------------


------------------------------------------------------------------------------
----------------------------------TaskReboot----------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:12 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:12 PM Client-Info Client OS:      Windows
 3/7/2018 12:12 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:12 PM Middleware::Response Module is disabled globally on the FOG server
------------------------------------------------------------------------------


------------------------------------------------------------------------------
--------------------------------HostnameChanger-------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:12 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:12 PM Client-Info Client OS:      Windows
 3/7/2018 12:12 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:12 PM Middleware::Response Success
 3/7/2018 12:12 PM HostnameChanger Checking Hostname
 3/7/2018 12:12 PM HostnameChanger Hostname is correct
 3/7/2018 12:12 PM HostnameChanger Attempting to join domain
 3/7/2018 12:12 PM HostnameChanger Host already joined to target domain
------------------------------------------------------------------------------


------------------------------------------------------------------------------
---------------------------------SnapinClient---------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:12 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:12 PM Client-Info Client OS:      Windows
 3/7/2018 12:12 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:12 PM Middleware::Response No snapins
------------------------------------------------------------------------------


------------------------------------------------------------------------------
--------------------------------PrinterManager--------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:12 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:12 PM Client-Info Client OS:      Windows
 3/7/2018 12:12 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:12 PM Middleware::Response Module is disabled on the host
------------------------------------------------------------------------------


------------------------------------------------------------------------------
--------------------------------PowerManagement-------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:12 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:12 PM Client-Info Client OS:      Windows
 3/7/2018 12:12 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:12 PM Middleware::Response Success
 3/7/2018 12:12 PM PowerManagement Calculating tasks to unschedule
 3/7/2018 12:12 PM PowerManagement Calculating tasks to schedule
------------------------------------------------------------------------------


------------------------------------------------------------------------------
----------------------------------UserTracker---------------------------------
------------------------------------------------------------------------------
 3/7/2018 12:12 PM Client-Info Client Version: 0.11.12
 3/7/2018 12:12 PM Client-Info Client OS:      Windows
 3/7/2018 12:12 PM Client-Info Server Version: 1.4.4
 3/7/2018 12:12 PM Middleware::Response Success
------------------------------------------------------------------------------

 3/7/2018 12:12 PM Service Sleeping for 1156 seconds
                          1 Reply Last reply Reply Quote 0
                          • J
                            Joe Schmitt Senior Developer
                            last edited by Joe Schmitt Mar 7, 2018, 12:30 PM Mar 7, 2018, 6:30 PM

                            Thread note: I will be in contact with @UWPVIOLATOR to see what works best when scaling for multiple clients, and/or if there are optimizations I could make to the client itself.

                            I am also building my own client stress-test environment so we can better replicate these issues.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                            1 Reply Last reply Reply Quote 0
                            • T
                              Tom Elliott
                              last edited by Mar 7, 2018, 9:22 PM

                              I was able to install PHP-FPM successfully and made the switchover to using it instead of apache’s libapache2-mod-php implementation (mod_prefork).

                              Load wise, the server is doing MUCH better. GUI is still slow though.

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              george1421G 1 Reply Last reply Mar 7, 2018, 9:32 PM Reply Quote 0
                              • george1421G
                                george1421 Moderator @Tom Elliott
                                last edited by george1421 Mar 7, 2018, 3:33 PM Mar 7, 2018, 9:32 PM

                                @tom-elliott Did you also switch over to memcached? or just phpfpm? I’m more curios for performance improvements than the OPs issue. (Oh did I say that on the outside…)

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                T 1 Reply Last reply Mar 7, 2018, 9:33 PM Reply Quote 0
                                • T
                                  Tom Elliott @george1421
                                  last edited by Mar 7, 2018, 9:33 PM

                                  @george1421 Just php-fpm.

                                  I don’t know how much help memcache would give, though I suppose it wouldn’t hurt to try it.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  george1421G 1 Reply Last reply Mar 7, 2018, 9:35 PM Reply Quote 1
                                  • george1421G
                                    george1421 Moderator @Tom Elliott
                                    last edited by george1421 Mar 7, 2018, 3:37 PM Mar 7, 2018, 9:35 PM

                                    @tom-elliott I need to look at the document to make sure its not capable of the zombie attach that impacted github, but here is what I setup before: https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast/3

                                    [edit] nevermind I already put in the bind to 127.0.0.1 in the tut so we are protected already [/edit]

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                    1 Reply Last reply Reply Quote 0
                                    • U
                                      UWPVIOLATOR
                                      last edited by Mar 8, 2018, 3:06 AM

                                      I will keep an eye on the server tomorrow let you guys know how goes.

                                      1 Reply Last reply Reply Quote 0
                                      • U
                                        UWPVIOLATOR
                                        last edited by Mar 8, 2018, 11:37 PM

                                        @Tom-Elliott @Joe-Schmitt

                                        Update

                                        Today was much better. The load stayed way down nothing like we have seen before so the fix is working in that aspect, but CPU utilization is still running close to 100% but never crashed like it did before. Because of the CPU usage the GUI was slow to respond. Did some talking today and we might just blow up all the images (we have 600 of them too, but not all of them don’t have image files on the server) and the database and start brand new when we move to the new server build. So I am going to see if we can quickly identify a good chunk of hosts we can remove in the meantime.

                                        @Tom-Elliott Our Sysadmin used PHPMyadmin to access the Mysql server and that was not working for him today. What do we need to do to get that to work with your fix from yesterday?

                                        T george1421G 2 Replies Last reply Mar 9, 2018, 12:18 AM Reply Quote 0
                                        • T
                                          Tom Elliott @UWPVIOLATOR
                                          last edited by Mar 9, 2018, 12:18 AM

                                          @uwpviolator we may have to manually install phpmyadmin. The one Ubuntu uses relies on libapache-mod-php so moving to php-fpm probably broke it from communicating. I can work with you tomorrow around 3pm if that works?

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                          U 1 Reply Last reply Mar 9, 2018, 1:02 AM Reply Quote 0
                                          • 1
                                          • 2
                                          • 3
                                          • 2 / 3
                                          • First post
                                            Last post

                                          204

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project