• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

After Update to SVN 4928 web pages time out constantly

Scheduled Pinned Locked Moved Solved
FOG Problems
4
55
33.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J
    Joseph Hales Testers
    last edited by Oct 13, 2015, 4:39 PM

    Old errors reoccur plus some new ones.

    
    [Tue Oct 13 11:32:00.141558 2015] [:error] [pid 3576] [client 10.110.50.40:65405] PHP Warning:  mysqli::query(): Couldn't fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
    [Tue Oct 13 11:32:01.035167 2015] [:error] [pid 3579] [client 10.117.50.11:53325] PHP Fatal error:  Uncaught exception 'ReflectionException' with message 'Class EventManager does not exist' in /var/www/html/fog/lib/fog/FOGBase.class.php:64\nStack trace:\n#0 /var/www/html/fog/lib/fog/FOGBase.class.php(64): ReflectionClass->__construct('EventManager')\n#1 /var/www/html/fog/commons/init.php(196): FOGBase->getClass('EventManager')\n#2 /var/www/html/fog/commons/base.inc.php(3): require_once('/var/www/html/f...')\n#3 /var/www/html/fog/service/servicemodule-active.php(2): require_once('/var/www/html/f...')\n#4 {main}\n  thrown in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
    [Tue Oct 13 11:32:01.089682 2015] [:error] [pid 3667] [client 10.1.50.220:53541] PHP Fatal error:  Uncaught exception 'ReflectionException' with message 'Class ServiceManager does not exist' in /var/www/html/fog/lib/fog/FOGBase.class.php:64\nStack trace:\n#0 /var/www/html/fog/lib/fog/FOGBase.class.php(64): ReflectionClass->__construct('ServiceManager')\n#1 /var/www/html/fog/lib/fog/FOGCore.class.php(14): FOGBase->getClass('ServiceManager')\n#2 /var/www/html/fog/lib/fog/FOGCore.class.php(217): FOGCore->getSetting('FOG_THEME')\n#3 /var/www/html/fog/commons/init.php(192): FOGCore->setSessionEnv()\n#4 /var/www/html/fog/commons/base.inc.php(3): require_once('/var/www/html/f...')\n#5 /var/www/html/fog/service/servicemodule-active.php(2): require_once('/var/www/html/f...')\n#6 {main}\n  thrown in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
    [Tue Oct 13 11:32:01.230274 2015] [:error] [pid 5538] [client 10.113.50.91:50426] PHP Fatal error:  Uncaught exception 'ReflectionException' with message 'Class ServiceManager does not exist' in /var/www/html/fog/lib/fog/FOGBase.class.php:64\nStack trace:\n#0 /var/www/html/fog/lib/fog/FOGBase.class.php(64): ReflectionClass->__construct('ServiceManager')\n#1 /var/www/html/fog/lib/fog/FOGCore.class.php(14): FOGBase->getClass('ServiceManager')\n#2 /var/www/html/fog/lib/fog/FOGCore.class.php(217): FOGCore->getSetting('FOG_THEME')\n#3 /var/www/html/fog/commons/init.php(192): FOGCore->setSessionEnv()\n#4 /var/www/html/fog/commons/base.inc.php(3): require_once('/var/www/html/f...')\n#5 /var/www/html/fog/service/snapins.checkin.php(2): require_once('/var/www/html/f...')\n#6 {main}\n  thrown in /var/www/html/fog/lib/fog/FOGBase.class.php on line 64
    [Tue Oct 13 11:34:49.125571 2015] [mpm_prefork:error] [pid 5892] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting```

    RTFM

    1 Reply Last reply Reply Quote 0
    • J
      Joseph Hales Testers
      last edited by Oct 13, 2015, 4:40 PM

      had to delete some of the errors as the post kept getting marked as spam by the filter but they seem to be repeats.

      RTFM

      W 1 Reply Last reply Oct 13, 2015, 5:46 PM Reply Quote 0
      • W
        Wayne Workman @Joseph Hales
        last edited by Oct 13, 2015, 5:46 PM

        @Joseph-Hales said:

        had to delete some of the errors as the post kept getting marked as spam by the filter but they seem to be repeats.

        You can get around that by uploading the logs in a .txt file.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
        Daily Clean Installation Results:
        https://fogtesting.fogproject.us/
        FOG Reporting:
        https://fog-external-reporting-results.fogproject.us/

        1 Reply Last reply Reply Quote 0
        • J
          Joseph Hales Testers
          last edited by Oct 13, 2015, 6:13 PM

          Ok yet another sign FOG hates me I come back from lunch my forum session wouldn’t let me log back in kept saying forbidden until I closed the window and reopened it 😉

          RTFM

          1 Reply Last reply Reply Quote 0
          • J
            Joseph Hales Testers
            last edited by Oct 13, 2015, 7:11 PM

            Should I roll back to SVN 4700 and if so what check out version is that?

            RTFM

            W 1 Reply Last reply Oct 13, 2015, 8:27 PM Reply Quote 0
            • W
              Wayne Workman @Joseph Hales
              last edited by Oct 13, 2015, 8:27 PM

              @Joseph-Hales if you go to your svn/trunk directory you can run svn info to get the revision number that you currently have checked out.

              You know the only safe method of downgrading is with a snapshot, right?

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
              Daily Clean Installation Results:
              https://fogtesting.fogproject.us/
              FOG Reporting:
              https://fog-external-reporting-results.fogproject.us/

              1 Reply Last reply Reply Quote 0
              • J
                Joseph Hales Testers
                last edited by Oct 13, 2015, 9:36 PM

                Any other suggestions it seems like after a restart sql hits some sort of loop state and just locks up with that apache maxworkers error.

                RTFM

                W 1 Reply Last reply Oct 13, 2015, 11:07 PM Reply Quote 0
                • W
                  Wayne Workman @Joseph Hales
                  last edited by Wayne Workman Oct 13, 2015, 5:19 PM Oct 13, 2015, 11:07 PM

                  @Joseph-Hales If it’s possible, disable the interface on the fog server at boot time.

                  On CentOS 7 / RHEL 7 / Fedora 19-22 you may do this by editing /etc/sysconfig/network-scripts/<interface name> and changing the line ONBOOT="yes" to ONBOOT="no"

                  http://ask.xmodulo.com/configure-static-ip-address-centos7.html

                  Then reboot and see how MySQL behaves… does it enter into a loop-like state then?

                  If not, it probably really is load related - and hopefully @Tom-Elliott can figure out what the difference is between then and now.

                  If it still has this problem with the interface turned off, then the problem resides on the fog server somewhere and is not load related.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                  Daily Clean Installation Results:
                  https://fogtesting.fogproject.us/
                  FOG Reporting:
                  https://fog-external-reporting-results.fogproject.us/

                  1 Reply Last reply Reply Quote 0
                  • T
                    Tom Elliott
                    last edited by Oct 14, 2015, 9:42 AM

                    I’ve completely rewritten session handling in the hopes that this would be fixed. I suspect that what’s happening is the sessions are being started and started many times. So after initial startup all the connections are in use for the new start up.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    1 Reply Last reply Reply Quote 1
                    • J
                      Joseph Hales Testers
                      last edited by Joseph Hales Oct 14, 2015, 9:48 AM Oct 14, 2015, 3:47 PM

                      I have increased MaxConnections and MaxRequestWorkers both to 1000 installed SVN 4956 and restarted the server with the exact same results now on the attached apache error log I notice it gets to the MaxRequestWorkers suggestion and then stops untill I restart the server. This behavior also appears in my earlier posts I just didn’t note it.
                      Apache.log.txt

                      RTFM

                      1 Reply Last reply Reply Quote 0
                      • J
                        Joseph Hales Testers
                        last edited by Oct 14, 2015, 4:08 PM

                        With the new settings I am occasionally able to get web interface pages to load 1 time out of 10 with several minutes to load each page but it doesn’t appear that Apache is logging anything anymore after it crashes on the max workers suggestion.

                        RTFM

                        1 Reply Last reply Reply Quote 0
                        • J
                          Joseph Hales Testers
                          last edited by Wayne Workman Oct 14, 2015, 1:04 PM Oct 14, 2015, 6:14 PM

                          Fog ping hosts seems to be my next resource hog after mysql is this normal?

                          TOP 10 MEM (ps auxww --sort=-rss|head -n10)
                          USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
                          mysql     1419  143  0.9 8723248 235708 ?      Ssl  10:14 252:42 /usr/sbin/mysqld
                          root      1956 32.5  0.5 312728 129040 ?       S    10:14  57:12 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
                          root      2027  0.0  0.1  90772 25100 ?        Ss   10:14   0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
                          root      2174  0.0  0.1 800960 24888 ?        Ssl  10:14   0:01 /opt/simpana/Base/cvd
                          sealion   1471  0.3  0.0 809564 22208 ?        Sl   10:14   0:38 python /usr/local/sealion-agent/etc/init.d/../../bin/sealion.py start
                          www-data  2695  0.3  0.0 328436 20700 ?        S    10:15   0:36 /usr/sbin/apache2 -k start
                          www-data  2359  0.2  0.0 329392 20344 ?        S    10:14   0:28 /usr/sbin/apache2 -k start
                          www-data  2702  0.2  0.0 402536 18924 ?        S    10:15   0:30 /usr/sbin/apache2 -k start
                          www-data  2299  0.2  0.0 475844 18440 ?        S    10:14   0:30 /usr/sbin/apache2 -k start
                          

                          Mod edited to use code box.

                          RTFM

                          T 1 Reply Last reply Oct 14, 2015, 10:32 PM Reply Quote 0
                          • J
                            Joseph Hales Testers
                            last edited by Oct 14, 2015, 7:50 PM

                            Here is what a fog client sees when the server Apache log is spamming the following.

                            fog.log

                            [Wed Oct 14 14:38:20.061493 2015] [:error] [pid 2706] [client 10.109.49.82:50064] PHP Warning:  mysqli::query(): Couldn't fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62

                            RTFM

                            1 Reply Last reply Reply Quote 0
                            • S
                              Sebastian Roth Moderator
                              last edited by Oct 14, 2015, 8:20 PM

                              Maybe your database is kind of crashed again?? https://forums.fogproject.org/topic/5764/fog-web-interface-times-out-loading-almost-any-page

                              Have you ever looked into mysql error logging?? May be start here: http://ronaldbradford.com/blog/monitoring-mysql-the-error-log-2009-09-16/ and http://www.pontikis.net/blog/how-and-when-to-enable-mysql-logs

                              If I got this right mysql should log errors to syslog in standard configuration. Not sure about that on all linux systems (RedHat vs. Debian vs. Arch vs. ???). But give it a try zgrep mysql /var/log/syslog* (output might be quite long!)

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              1 Reply Last reply Reply Quote 0
                              • T
                                Tom Elliott @Joseph Hales
                                last edited by Oct 14, 2015, 10:32 PM

                                @Joseph-Hales ping hosts could be a culprit in that it pings all of your hosts as a service separately from the main. The ping time happens every five minutes after the last completion. My guess is this IS the sole reason to the problem you’re seeing considering from what I recall you have about 6k hosts. You can disable the service and my guess is you’ll see much better stays overall on your server, including MySQL.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                1 Reply Last reply Reply Quote 1
                                • T
                                  Tom Elliott
                                  last edited by Oct 14, 2015, 10:45 PM

                                  More specifically.

                                  FOGPingHosts works as a service similar to FOGMulticastManager or FOGImageReplicator. It’s purpose is to ping the hosts and report the “status” so that you’re not waiting on the List All/Search on the GUI to move between pages.

                                  It pings each host and updates the ping status column on the database. This iterates every 5 minutes after the last one updates. So every 5 minutes it’s pinging all hosts of the server.

                                  Now I’ve added a usleep of 1/1000th of a second per each host ping to allow the CPU to cool down between themselves so we don’t completely overload your CPU. I’ll up this value just to help slow things down a bit to 1/10th of a second. I’m not going to adjust the sleep time as 5 minutes is okay for most, though you can adjust your value in the Config.class.php.

                                  This would explain the mysql usage AND the FOGPingHosts taking up most of the time.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  1 Reply Last reply Reply Quote 1
                                  • J
                                    Joseph Hales Testers
                                    last edited by Oct 15, 2015, 12:31 AM

                                    I updated to the latest SVN same issue after restart where would I find Config.Class.php and what would I edit to remove ping hosts entirely?

                                    RTFM

                                    T W 2 Replies Last reply Oct 15, 2015, 12:37 AM Reply Quote 0
                                    • T
                                      Tom Elliott @Joseph Hales
                                      last edited by Oct 15, 2015, 12:37 AM

                                      @Joseph-Hales what version? Do you have ping on?

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                      1 Reply Last reply Reply Quote 1
                                      • W
                                        Wayne Workman @Joseph Hales
                                        last edited by Oct 15, 2015, 12:40 AM

                                        @Joseph-Hales

                                        See my awkwardly placed comment here?

                                        /var/www/html/fog/lib/service/PingHosts.class.php

                                        #                $this->getClass(Host,$Host)->set(pingstatus,(int)$this->getClass(Ping,$ip)->execute())->save();
                                        
                                            public $zzz = PINGHOSTSLEEPTIME;
                                            private function commonOutput() {
                                                try {
                                                    $webServerIP = $this->FOGCore->resolveHostName($this->FOGCore->getSetting('FOG_WEB_HOST'));
                                                    $this->outall(sprintf(' * FOG Web Host IP: %s',$webServerIP));
                                                    $serverIPs = $this->getIPAddress();
                                                    foreach ((array)$serverIPs AS $i => &$ip) {
                                                        if (!$i) $this->outall(" * This server's IP Addresses");
                                                        $this->outall(" |\t$ip");
                                                    }
                                                    if (!in_array($this->FOGCore->resolveHostname($this->FOGCore->getSetting(FOG_WEB_HOST)),$this->getIPAddress())) throw new Exception(_('I am not the fog web server'));
                                                    $this->outall(' * Attempting to ping '.$this->getClass(HostManager)->count().' host(s).');
                                                    $Hosts = $this->getClass(HostManager)->find('','','','','','','','id');
                                                    foreach ($Hosts AS $i => &$Host) {
                                                        // Ensures the hostIP regardless of how it is entered,
                                                        // to remove any beginning/ending white space
                                                        $hostIP = trim($this->getClass(Host,$Host)->get(ip));
                                                        // Test IP Value and if valid, use it as the pinging source
                                                        if (filter_var($hostIP,FILTER_VALIDATE_IP)) $ip = $hostIP;
                                                        // Otherwise attempt to get the hostname resolved.
                                                        else $ip = $this->FOGCore->resolveHostname($this->getClass(Host,$Host)->get(name));
                                                        // If the host still isn't found, set value to -1
                                                        // Allows us to clarify what is up.
                                                        if (!filter_var($ip,FILTER_VALIDATE_IP)) {
                                                            $this->getClass(Host,$Host)->set(pingstatus,-1)->save();
                                                            continue;
                                                        }
                                                        // If all above makes it here, perform the ping
                                        #                $this->getClass(Host,$Host)->set(pingstatus,(int)$this->getClass(Ping,$ip)->execute())->save();
                                                        // Give CPU a little breather between pings
                                                        usleep(1000);
                                                    }
                                                    unset($Host);
                                                    $this->outall(' * All status\' have been updated');
                                                } catch (Exception $e) {
                                                    $this->outall($e->getMessage());
                                                }
                                            }
                                            public function serviceRun() {
                                                $this->out(' ',$this->dev);
                                                $this->out(' +---------------------------------------------------------',$this->dev);
                                                $this->commonOutput();
                                                $this->out(' +---------------------------------------------------------',$this->dev);
                                            }
                                        }
                                        
                                        

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                        Daily Clean Installation Results:
                                        https://fogtesting.fogproject.us/
                                        FOG Reporting:
                                        https://fog-external-reporting-results.fogproject.us/

                                        1 Reply Last reply Reply Quote 0
                                        • W
                                          Wayne Workman
                                          last edited by Oct 15, 2015, 12:42 AM

                                          Probably a lot better job could be done there…

                                          Like commenting out the entire loop.

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                          Daily Clean Installation Results:
                                          https://fogtesting.fogproject.us/
                                          FOG Reporting:
                                          https://fog-external-reporting-results.fogproject.us/

                                          1 Reply Last reply Reply Quote 0
                                          • 1
                                          • 2
                                          • 3
                                          • 2 / 3
                                          2 / 3
                                          • First post
                                            32/55
                                            Last post

                                          223

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project