After Update to SVN 4928 web pages time out constantly
-
@Joseph-Hales if you go to your svn/trunk directory you can run
svn infoto get the revision number that you currently have checked out.You know the only safe method of downgrading is with a snapshot, right?
-
Any other suggestions it seems like after a restart sql hits some sort of loop state and just locks up with that apache maxworkers error.
-
@Joseph-Hales If it’s possible, disable the interface on the fog server at boot time.
On CentOS 7 / RHEL 7 / Fedora 19-22 you may do this by editing
/etc/sysconfig/network-scripts/<interface name>and changing the lineONBOOT="yes"toONBOOT="no"http://ask.xmodulo.com/configure-static-ip-address-centos7.html
Then reboot and see how MySQL behaves… does it enter into a loop-like state then?
If not, it probably really is load related - and hopefully @Tom-Elliott can figure out what the difference is between then and now.
If it still has this problem with the interface turned off, then the problem resides on the fog server somewhere and is not load related.
-
I’ve completely rewritten session handling in the hopes that this would be fixed. I suspect that what’s happening is the sessions are being started and started many times. So after initial startup all the connections are in use for the new start up.
-
I have increased MaxConnections and MaxRequestWorkers both to 1000 installed SVN 4956 and restarted the server with the exact same results now on the attached apache error log I notice it gets to the MaxRequestWorkers suggestion and then stops untill I restart the server. This behavior also appears in my earlier posts I just didn’t note it.
Apache.log.txt -
With the new settings I am occasionally able to get web interface pages to load 1 time out of 10 with several minutes to load each page but it doesn’t appear that Apache is logging anything anymore after it crashes on the max workers suggestion.
-
Fog ping hosts seems to be my next resource hog after mysql is this normal?
TOP 10 MEM (ps auxww --sort=-rss|head -n10) USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND mysql 1419 143 0.9 8723248 235708 ? Ssl 10:14 252:42 /usr/sbin/mysqld root 1956 32.5 0.5 312728 129040 ? S 10:14 57:12 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts root 2027 0.0 0.1 90772 25100 ? Ss 10:14 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf root 2174 0.0 0.1 800960 24888 ? Ssl 10:14 0:01 /opt/simpana/Base/cvd sealion 1471 0.3 0.0 809564 22208 ? Sl 10:14 0:38 python /usr/local/sealion-agent/etc/init.d/../../bin/sealion.py start www-data 2695 0.3 0.0 328436 20700 ? S 10:15 0:36 /usr/sbin/apache2 -k start www-data 2359 0.2 0.0 329392 20344 ? S 10:14 0:28 /usr/sbin/apache2 -k start www-data 2702 0.2 0.0 402536 18924 ? S 10:15 0:30 /usr/sbin/apache2 -k start www-data 2299 0.2 0.0 475844 18440 ? S 10:14 0:30 /usr/sbin/apache2 -k startMod edited to use code box.
RTFM
-
Here is what a fog client sees when the server Apache log is spamming the following.
[Wed Oct 14 14:38:20.061493 2015] [:error] [pid 2706] [client 10.109.49.82:50064] PHP Warning: mysqli::query(): Couldn't fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62 -
Maybe your database is kind of crashed again?? https://forums.fogproject.org/topic/5764/fog-web-interface-times-out-loading-almost-any-page
Have you ever looked into mysql error logging?? May be start here: http://ronaldbradford.com/blog/monitoring-mysql-the-error-log-2009-09-16/ and http://www.pontikis.net/blog/how-and-when-to-enable-mysql-logs
If I got this right mysql should log errors to syslog in standard configuration. Not sure about that on all linux systems (RedHat vs. Debian vs. Arch vs. ???). But give it a try
zgrep mysql /var/log/syslog*(output might be quite long!) -
@Joseph-Hales ping hosts could be a culprit in that it pings all of your hosts as a service separately from the main. The ping time happens every five minutes after the last completion. My guess is this IS the sole reason to the problem you’re seeing considering from what I recall you have about 6k hosts. You can disable the service and my guess is you’ll see much better stays overall on your server, including MySQL.
-
More specifically.
FOGPingHosts works as a service similar to FOGMulticastManager or FOGImageReplicator. It’s purpose is to ping the hosts and report the “status” so that you’re not waiting on the List All/Search on the GUI to move between pages.
It pings each host and updates the ping status column on the database. This iterates every 5 minutes after the last one updates. So every 5 minutes it’s pinging all hosts of the server.
Now I’ve added a usleep of 1/1000th of a second per each host ping to allow the CPU to cool down between themselves so we don’t completely overload your CPU. I’ll up this value just to help slow things down a bit to 1/10th of a second. I’m not going to adjust the sleep time as 5 minutes is okay for most, though you can adjust your value in the Config.class.php.
This would explain the mysql usage AND the FOGPingHosts taking up most of the time.
-
I updated to the latest SVN same issue after restart where would I find Config.Class.php and what would I edit to remove ping hosts entirely?
RTFM
-
@Joseph-Hales what version? Do you have ping on?
-
See my awkwardly placed comment here?
/var/www/html/fog/lib/service/PingHosts.class.php# $this->getClass(Host,$Host)->set(pingstatus,(int)$this->getClass(Ping,$ip)->execute())->save();public $zzz = PINGHOSTSLEEPTIME; private function commonOutput() { try { $webServerIP = $this->FOGCore->resolveHostName($this->FOGCore->getSetting('FOG_WEB_HOST')); $this->outall(sprintf(' * FOG Web Host IP: %s',$webServerIP)); $serverIPs = $this->getIPAddress(); foreach ((array)$serverIPs AS $i => &$ip) { if (!$i) $this->outall(" * This server's IP Addresses"); $this->outall(" |\t$ip"); } if (!in_array($this->FOGCore->resolveHostname($this->FOGCore->getSetting(FOG_WEB_HOST)),$this->getIPAddress())) throw new Exception(_('I am not the fog web server')); $this->outall(' * Attempting to ping '.$this->getClass(HostManager)->count().' host(s).'); $Hosts = $this->getClass(HostManager)->find('','','','','','','','id'); foreach ($Hosts AS $i => &$Host) { // Ensures the hostIP regardless of how it is entered, // to remove any beginning/ending white space $hostIP = trim($this->getClass(Host,$Host)->get(ip)); // Test IP Value and if valid, use it as the pinging source if (filter_var($hostIP,FILTER_VALIDATE_IP)) $ip = $hostIP; // Otherwise attempt to get the hostname resolved. else $ip = $this->FOGCore->resolveHostname($this->getClass(Host,$Host)->get(name)); // If the host still isn't found, set value to -1 // Allows us to clarify what is up. if (!filter_var($ip,FILTER_VALIDATE_IP)) { $this->getClass(Host,$Host)->set(pingstatus,-1)->save(); continue; } // If all above makes it here, perform the ping # $this->getClass(Host,$Host)->set(pingstatus,(int)$this->getClass(Ping,$ip)->execute())->save(); // Give CPU a little breather between pings usleep(1000); } unset($Host); $this->outall(' * All status\' have been updated'); } catch (Exception $e) { $this->outall($e->getMessage()); } } public function serviceRun() { $this->out(' ',$this->dev); $this->out(' +---------------------------------------------------------',$this->dev); $this->commonOutput(); $this->out(' +---------------------------------------------------------',$this->dev); } } -
Probably a lot better job could be done there…
Like commenting out the entire loop.
-
Latest version of SVN 4162 did remove host ping from my top 10 loads but did not fix the issue with the errors we were seeing and web pages not responding. MySQLCheck did not find any issues with the database itself.
-
I think that these are to blame.
/var/www/html/fog/lib/fog/FOGCore.class.phppublic function cleanInvalidEntries() { // $this->DB->query("DELETE FROM aloLog WHERE alID=0"); // $this->DB->query("DELETE FROM clientUpdates WHERE cuID=0"); // $this->DB->query("DELETE FROM dirCleaner WHERE dcID=0"); // $this->DB->query("DELETE FROM globalSettings WHERE settingID=0"); // $this->DB->query("DELETE FROM greenFog WHERE gfID=0"); // $this->DB->query("DELETE FROM groupMembers WHERE gmID=0 OR gmHostID=0 OR gmGroupID=0"); // $this->DB->query("DELETE FROM groups WHERE groupID=0"); // $this->DB->query("DELETE FROM history WHERE hID=0"); // $this->DB->query("DELETE FROM hostAutoLogOut WHERE haloID=0 OR haloHostID=0"); // $this->DB->query("DELETE FROM hostMAC where hmID=0 OR hmHostID=0"); // $this->DB->query("DELETE FROM hosts WHERE hostID=0"); // $this->DB->query("DELETE FROM hostScreenSettings WHERE hssID=0 OR hssHostID=0"); // $this->DB->query("DELETE FROM imageGroupAssoc WHERE igaID=0 OR igaImageID=0 OR igaStorageGroupID=0"); // $this->DB->query("DELETE FROM imagePartitionTypes WHERE imagePartitionTypeID=0"); // $this->DB->query("DELETE FROM images WHERE imageID=0"); // $this->DB->query("DELETE FROM imageTypes WHERE imageTypeID=0"); // $this->DB->query("DELETE FROM imagingLog WHERE ilID=0 OR ilHostID=0"); // $this->DB->query("DELETE FROM inventory WHERE iID=0 OR iHostID=0"); // $this->DB->query("DELETE FROM ipxeTable WHERE ipxeID=0"); // $this->DB->query("DELETE FROM keySequence WHERE ksID=0"); // $this->DB->query("DELETE FROM modules WHERE id=0"); // $this->DB->query("DELETE FROM moduleStatusByHost WHERE msID=0 OR msHostID=0 OR msModuleID=0"); // $this->DB->query("DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0"); // $this->DB->query("DELETE FROM multicastSessionsAssoc WHERE msaID=0 OR msID=0 OR tID=0"); // $this->DB->query("DELETE FROM nfsFailures WHERE nfID=0 OR nfNodeID=0 OR nfTaskID=0 OR nfHostID=0 OR nfGroupID=0"); // $this->DB->query("DELETE FROM nfsGroupMembers WHERE ngmID=0 OR ngmGroupID=0"); // $this->DB->query("DELETE FROM nfsGroups WHERE ngID=0"); // $this->DB->query("DELETE FROM os WHERE osID=0"); // $this->DB->query("DELETE FROM oui WHERE ouiID=0"); // $this->DB->query("DELETE FROM peer WHERE id=0"); // $this->DB->query("DELETE FROM peer_torrent WHERE id=0 OR peer_id=0 OR torrent_id=0"); // $this->DB->query("DELETE FROM plugins WHERE pID=0"); // $this->DB->query("DELETE FROM printerAssoc WHERE paID=0 OR paHostID=0 OR paPrinterID=0"); // $this->DB->query("DELETE FROM printers WHERE pID=0"); // $this->DB->query("DELETE FROM pushbullet WHERE pID=0"); // $this->DB->query("DELETE FROM pxeMenu WHERE pxeID=0"); // $this->DB->query("DELETE FROM scheduledTasks WHERE stID=0 OR stTaskTypeID=0 OR stGroupHostID=0 OR stActive=0"); // $this->DB->query("DELETE FROM schemaVersion WHERE vID=0"); // $this->DB->query("DELETE FROM snapinAssoc WHERE saID=0 OR saHostID=0 OR saSnapinID=0"); // $this->DB->query("DELETE FROM snapinGroupAssoc WHERE sgaID=0 OR sgaSnapinID=0 OR sgaStorageGroupID=0"); // $this->DB->query("DELETE FROM snapinJobs WHERE sjID=0 OR sjHostID=0"); // $this->DB->query("DELETE FROM snapins WHERE snapinID=0"); // $this->DB->query("DELETE FROM snapinTasks WHERE stID=0 OR stJobID=0 OR stSnapinID=0"); // $this->DB->query("DELETE FROM supportedOS WHERE osID=0"); // $this->DB->query("DELETE FROM taskLog WHERE id=0 OR taskID=0"); // $this->DB->query("DELETE FROM tasks WHERE taskID=0 OR taskHostID=0 OR taskTypeID=0"); // $this->DB->query("DELETE FROM taskStates WHERE tsID=0"); // $this->DB->query("DELETE FROM taskTypes WHERE ttID=0"); // $this->DB->query("DELETE FROM torrent WHERE id=0"); // $this->DB->query("DELETE FROM userCleanup WHERE ucID=0"); }I don’t think innodb can handle this big loop over and over.
This is the stuff that was showing on Joseph’s system over and over when he ran this command against mysql:
show full processlist;mysql> show full processlist; ±----±-----±----------------±-----±--------±-----±-----------------------------±-------------------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | ±----±-----±----------------±-----±--------±-----±-----------------------------±-------------------------------------------------------------+ | 172 | root | localhost:53903 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 173 | root | localhost:53904 | fog | Query | 124 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 174 | root | localhost:53905 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 175 | root | localhost:53907 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 176 | root | localhost:53906 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 177 | root | localhost:53909 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 178 | root | localhost:53910 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 179 | root | localhost:53911 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 180 | root | localhost:53912 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 181 | root | localhost:53913 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 182 | root | localhost:53914 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 183 | root | localhost:53915 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 184 | root | localhost:53916 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 185 | root | localhost:53917 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 186 | root | localhost:53918 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 187 | root | localhost:53919 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 188 | root | localhost:53920 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 189 | root | localhost:53921 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 190 | root | localhost:53922 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 191 | root | localhost:53923 | fog | Query | 124 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 192 | root | localhost:53924 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 193 | root | localhost:53925 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 194 | root | localhost:53926 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 195 | root | localhost:53927 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 196 | root | localhost:53928 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 197 | root | localhost:53929 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 198 | root | localhost:53930 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 199 | root | localhost:53932 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 200 | root | localhost:53933 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 201 | root | localhost:53934 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 202 | root | localhost:53935 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 203 | root | localhost:53937 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 204 | root | localhost:53936 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 205 | root | localhost:53939 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 206 | root | localhost:53941 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 207 | root | localhost:53940 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 208 | root | localhost:53943 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 209 | root | localhost:53945 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 210 | root | localhost:53944 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 211 | root | localhost:53947 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 212 | root | localhost:53948 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 213 | root | localhost:53949 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 214 | root | localhost:53950 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 215 | root | localhost:53951 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 216 | root | localhost:53952 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 217 | root | localhost:53953 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 218 | root | localhost:53954 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 219 | root | localhost:53955 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 220 | root | localhost:53956 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 221 | root | localhost:53957 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 222 | root | localhost:53958 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 223 | root | localhost:53959 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 224 | root | localhost:53960 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 225 | root | localhost:53961 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 226 | root | localhost:53962 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 227 | root | localhost:53963 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 228 | root | localhost:53964 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 229 | root | localhost:53965 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 230 | root | localhost:53966 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 231 | root | localhost:53967 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 232 | root | localhost:53968 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 233 | root | localhost:53969 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 234 | root | localhost:53970 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 235 | root | localhost:53971 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 236 | root | localhost:53972 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 237 | root | localhost:53973 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 238 | root | localhost:53974 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 239 | root | localhost:53976 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 240 | root | localhost:53977 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 241 | root | localhost:53978 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 242 | root | localhost:53979 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 243 | root | localhost:53980 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 244 | root | localhost:53981 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 245 | root | localhost:53982 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 246 | root | localhost:53983 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 247 | root | localhost:53984 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 248 | root | localhost:53985 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 249 | root | localhost:53986 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 250 | root | localhost:53987 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 251 | root | localhost:53988 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 252 | root | localhost:53989 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 253 | root | localhost:53990 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 254 | root | localhost:53991 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 255 | root | localhost:53992 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 256 | root | localhost:53993 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 257 | root | localhost:53994 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 258 | root | localhost:53996 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 259 | root | localhost:53997 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 260 | root | localhost:53998 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 261 | root | localhost:53999 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 262 | root | localhost:54000 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 263 | root | localhost:54001 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 264 | root | localhost:54002 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 265 | root | localhost:54003 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 266 | root | localhost:54004 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 267 | root | localhost:54005 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 268 | root | localhost:54006 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 269 | root | localhost:54007 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 270 | root | localhost:54008 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 271 | root | localhost:54009 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 272 | root | localhost:54011 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 273 | root | localhost:54012 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 274 | root | localhost:54014 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 275 | root | localhost:54013 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 276 | root | localhost:54016 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 277 | root | localhost:54017 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 278 | root | localhost:54018 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 279 | root | localhost:54019 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 280 | root | localhost:54020 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 281 | root | localhost:54021 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 282 | root | localhost:54022 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 283 | root | localhost:54023 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 284 | root | localhost:54024 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 285 | root | localhost:54025 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 286 | root | localhost:54026 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 287 | root | localhost:54027 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 288 | root | localhost:54028 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 289 | root | localhost:54029 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 290 | root | localhost:54030 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 291 | root | localhost:54031 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 292 | root | localhost:54032 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 293 | root | localhost:54033 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 294 | root | localhost:54034 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 295 | root | localhost:54035 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 296 | root | localhost:54036 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 297 | root | localhost:54037 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 298 | root | localhost:54038 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 299 | root | localhost:54039 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 300 | root | localhost:54040 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 301 | root | localhost:54042 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 302 | root | localhost:54041 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 303 | root | localhost:54044 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 304 | root | localhost:54045 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 305 | root | localhost:54046 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 306 | root | localhost:54047 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 307 | root | localhost:54048 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 308 | root | localhost:54049 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 309 | root | localhost:54050 | fog | Query | 124 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 310 | root | localhost:54051 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 311 | root | localhost:54052 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 312 | root | localhost:54053 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 313 | root | localhost:54054 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 314 | root | localhost:54055 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 315 | root | localhost:54056 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 316 | root | localhost:54057 | fog | Query | 124 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 317 | root | localhost:54058 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 318 | root | localhost:54059 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 319 | root | localhost:54060 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 320 | root | localhost:54061 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 321 | root | localhost:54062 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 322 | root | localhost:54063 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 323 | root | localhost:54064 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 324 | root | localhost:54065 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 325 | root | localhost:54067 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 326 | root | localhost:54066 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 327 | root | localhost:54068 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 328 | root | localhost:54072 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 329 | root | localhost:54073 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 330 | root | localhost:54074 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 331 | root | localhost:54075 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 332 | root | localhost:54077 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 333 | root | localhost:54078 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 334 | root | localhost:54079 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 335 | root | localhost:54080 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 336 | root | localhost:54081 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 337 | root | localhost:54082 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 338 | root | localhost:54083 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 339 | root | localhost:54084 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 340 | root | localhost:54086 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 341 | root | localhost:54085 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 342 | root | localhost:54088 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 343 | root | localhost:54089 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 344 | root | localhost:54090 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 345 | root | localhost:54091 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 346 | root | localhost:54092 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 347 | root | localhost:54093 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 348 | root | localhost:54094 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 349 | root | localhost:54095 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 350 | root | localhost:54096 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 351 | root | localhost:54097 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 352 | root | localhost:54098 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 353 | root | localhost:54099 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 354 | root | localhost:54100 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 355 | root | localhost:54101 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 356 | root | localhost:54102 | fog | Query | 124 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 357 | root | localhost:54103 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 358 | root | localhost:54104 | fog | Query | 124 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 359 | root | localhost:54105 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 360 | root | localhost:54106 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 361 | root | localhost:54107 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 362 | root | localhost:54108 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 363 | root | localhost:54109 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 364 | root | localhost:54110 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 365 | root | localhost:54111 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 366 | root | localhost:54112 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 367 | root | localhost:54113 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 368 | root | localhost:54114 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 369 | root | localhost:54115 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 370 | root | localhost:54116 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 | | 371 | root | localhost:54117 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 | | 372 | root | localhost:54119 | fog | Sleep | 19 | | NULL | | 373 | root | localhost:54120 | fog | Sleep | 6 | | NULL | | 374 | root | localhost:54121 | fog | Sleep | 19 | | NULL | | 375 | root | localhost:54122 | fog | Sleep | 19 | | NULL | | 376 | root | localhost:54123 | fog | Sleep | 19 | | NULL | | 380 | root | localhost | fog | Query | 0 | NULL | show full processlist | ±----±-----±----------------±-----±--------±-----±-----------------------------±-------------------------------------------------------------+ 206 rows in set (0.00 sec) -
Removed that method and caller so maybe now it’ll be better.
-
The web page seems to be working now but I am still having some issues and the Apache stops logging a min after restart with the same errors as shown in the attached log.
Apache.log.txt -
@Joseph-Hales I wonder if the sheer amount of errors being recorded simply crash the logging mechanism… ?
There are levels of logging in daemons, you can adjust the level on a scale, some have different modes.
This was written for Ubuntu, but I would imagine that the things in here can be applied to any distro very similarly:
https://www.digitalocean.com/community/tutorials/how-to-configure-logging-and-log-rotation-in-apache-on-an-ubuntu-vpsIt might be beneficial to turn up the verbosity of logging just to figure out what’s going on… and it might be beneficial to turn the verbosity down to lower cpu load … and so logging doesn’t crash.
