After Update to SVN 4928 web pages time out constantly

Tom Elliott

@Joseph-Hales what version? Do you have ping on?

Wayne Workman

See my awkwardly placed comment here?

/var/www/html/fog/lib/service/PingHosts.class.php

#                $this->getClass(Host,$Host)->set(pingstatus,(int)$this->getClass(Ping,$ip)->execute())->save();

    public $zzz = PINGHOSTSLEEPTIME;
    private function commonOutput() {
        try {
            $webServerIP = $this->FOGCore->resolveHostName($this->FOGCore->getSetting('FOG_WEB_HOST'));
            $this->outall(sprintf(' * FOG Web Host IP: %s',$webServerIP));
            $serverIPs = $this->getIPAddress();
            foreach ((array)$serverIPs AS $i => &$ip) {
                if (!$i) $this->outall(" * This server's IP Addresses");
                $this->outall(" |\t$ip");
            }
            if (!in_array($this->FOGCore->resolveHostname($this->FOGCore->getSetting(FOG_WEB_HOST)),$this->getIPAddress())) throw new Exception(_('I am not the fog web server'));
            $this->outall(' * Attempting to ping '.$this->getClass(HostManager)->count().' host(s).');
            $Hosts = $this->getClass(HostManager)->find('','','','','','','','id');
            foreach ($Hosts AS $i => &$Host) {
                // Ensures the hostIP regardless of how it is entered,
                // to remove any beginning/ending white space
                $hostIP = trim($this->getClass(Host,$Host)->get(ip));
                // Test IP Value and if valid, use it as the pinging source
                if (filter_var($hostIP,FILTER_VALIDATE_IP)) $ip = $hostIP;
                // Otherwise attempt to get the hostname resolved.
                else $ip = $this->FOGCore->resolveHostname($this->getClass(Host,$Host)->get(name));
                // If the host still isn't found, set value to -1
                // Allows us to clarify what is up.
                if (!filter_var($ip,FILTER_VALIDATE_IP)) {
                    $this->getClass(Host,$Host)->set(pingstatus,-1)->save();
                    continue;
                }
                // If all above makes it here, perform the ping
#                $this->getClass(Host,$Host)->set(pingstatus,(int)$this->getClass(Ping,$ip)->execute())->save();
                // Give CPU a little breather between pings
                usleep(1000);
            }
            unset($Host);
            $this->outall(' * All status\' have been updated');
        } catch (Exception $e) {
            $this->outall($e->getMessage());
        }
    }
    public function serviceRun() {
        $this->out(' ',$this->dev);
        $this->out(' +---------------------------------------------------------',$this->dev);
        $this->commonOutput();
        $this->out(' +---------------------------------------------------------',$this->dev);
    }
}

Wayne Workman

Probably a lot better job could be done there…

Like commenting out the entire loop.

Joseph Hales

Latest version of SVN 4162 did remove host ping from my top 10 loads but did not fix the issue with the errors we were seeing and web pages not responding. MySQLCheck did not find any issues with the database itself.

Wayne Workman

I think that these are to blame.

/var/www/html/fog/lib/fog/FOGCore.class.php

public function cleanInvalidEntries() {
   //     $this->DB->query("DELETE FROM aloLog WHERE alID=0");
   //    $this->DB->query("DELETE FROM clientUpdates WHERE cuID=0");
  //      $this->DB->query("DELETE FROM dirCleaner WHERE dcID=0");
  //      $this->DB->query("DELETE FROM globalSettings WHERE settingID=0");
  //      $this->DB->query("DELETE FROM greenFog WHERE gfID=0");
 //       $this->DB->query("DELETE FROM groupMembers WHERE gmID=0 OR gmHostID=0 OR gmGroupID=0");
 //       $this->DB->query("DELETE FROM groups WHERE groupID=0");
 //       $this->DB->query("DELETE FROM history WHERE hID=0");
 //       $this->DB->query("DELETE FROM hostAutoLogOut WHERE haloID=0 OR haloHostID=0");
 //       $this->DB->query("DELETE FROM hostMAC where hmID=0 OR hmHostID=0");
 //       $this->DB->query("DELETE FROM hosts WHERE hostID=0");
 //       $this->DB->query("DELETE FROM hostScreenSettings WHERE hssID=0 OR hssHostID=0");
 //       $this->DB->query("DELETE FROM imageGroupAssoc WHERE igaID=0 OR igaImageID=0 OR igaStorageGroupID=0");
 //       $this->DB->query("DELETE FROM imagePartitionTypes WHERE imagePartitionTypeID=0");
 //       $this->DB->query("DELETE FROM images WHERE imageID=0");
 //       $this->DB->query("DELETE FROM imageTypes WHERE imageTypeID=0");
 //       $this->DB->query("DELETE FROM imagingLog WHERE ilID=0 OR ilHostID=0");
 //       $this->DB->query("DELETE FROM inventory WHERE iID=0 OR iHostID=0");
 //       $this->DB->query("DELETE FROM ipxeTable WHERE ipxeID=0");
//       $this->DB->query("DELETE FROM keySequence WHERE ksID=0");
//        $this->DB->query("DELETE FROM modules WHERE id=0");
 //       $this->DB->query("DELETE FROM moduleStatusByHost WHERE msID=0 OR msHostID=0 OR msModuleID=0");
//        $this->DB->query("DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0");
//        $this->DB->query("DELETE FROM multicastSessionsAssoc WHERE msaID=0 OR msID=0 OR tID=0");
//        $this->DB->query("DELETE FROM nfsFailures WHERE nfID=0 OR nfNodeID=0 OR nfTaskID=0 OR nfHostID=0 OR nfGroupID=0");
//        $this->DB->query("DELETE FROM nfsGroupMembers WHERE ngmID=0 OR ngmGroupID=0");
//        $this->DB->query("DELETE FROM nfsGroups WHERE ngID=0");
//        $this->DB->query("DELETE FROM os WHERE osID=0");
//        $this->DB->query("DELETE FROM oui WHERE ouiID=0");
//        $this->DB->query("DELETE FROM peer WHERE id=0");
//        $this->DB->query("DELETE FROM peer_torrent WHERE id=0 OR peer_id=0 OR torrent_id=0");
//        $this->DB->query("DELETE FROM plugins WHERE pID=0");
//        $this->DB->query("DELETE FROM printerAssoc WHERE paID=0 OR paHostID=0 OR paPrinterID=0");
//        $this->DB->query("DELETE FROM printers WHERE pID=0");
//        $this->DB->query("DELETE FROM pushbullet WHERE pID=0");
//        $this->DB->query("DELETE FROM pxeMenu WHERE pxeID=0");
//        $this->DB->query("DELETE FROM scheduledTasks WHERE stID=0 OR stTaskTypeID=0 OR stGroupHostID=0 OR stActive=0");
 //       $this->DB->query("DELETE FROM schemaVersion WHERE vID=0");
 //       $this->DB->query("DELETE FROM snapinAssoc WHERE saID=0 OR saHostID=0 OR saSnapinID=0");
//        $this->DB->query("DELETE FROM snapinGroupAssoc WHERE sgaID=0 OR sgaSnapinID=0 OR sgaStorageGroupID=0");
//        $this->DB->query("DELETE FROM snapinJobs WHERE sjID=0 OR sjHostID=0");
//        $this->DB->query("DELETE FROM snapins WHERE snapinID=0");
//        $this->DB->query("DELETE FROM snapinTasks WHERE stID=0 OR stJobID=0 OR stSnapinID=0");
//        $this->DB->query("DELETE FROM supportedOS WHERE osID=0");
 //       $this->DB->query("DELETE FROM taskLog WHERE id=0 OR taskID=0");
 //       $this->DB->query("DELETE FROM tasks WHERE taskID=0 OR taskHostID=0 OR taskTypeID=0");
 //       $this->DB->query("DELETE FROM taskStates WHERE tsID=0");
 //       $this->DB->query("DELETE FROM taskTypes WHERE ttID=0");
 //       $this->DB->query("DELETE FROM torrent WHERE id=0");
 //       $this->DB->query("DELETE FROM userCleanup WHERE ucID=0");
    }

I don’t think innodb can handle this big loop over and over.

This is the stuff that was showing on Joseph’s system over and over when he ran this command against mysql:
show full processlist;

mysql> show full processlist;
±----±-----±----------------±-----±--------±-----±-----------------------------±-------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
±----±-----±----------------±-----±--------±-----±-----------------------------±-------------------------------------------------------------+
| 172 | root | localhost:53903 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 173 | root | localhost:53904 | fog | Query | 124 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 174 | root | localhost:53905 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 175 | root | localhost:53907 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 176 | root | localhost:53906 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 177 | root | localhost:53909 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 178 | root | localhost:53910 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 179 | root | localhost:53911 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 180 | root | localhost:53912 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 181 | root | localhost:53913 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 182 | root | localhost:53914 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 183 | root | localhost:53915 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 184 | root | localhost:53916 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 185 | root | localhost:53917 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 186 | root | localhost:53918 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 187 | root | localhost:53919 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 188 | root | localhost:53920 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 189 | root | localhost:53921 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 190 | root | localhost:53922 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 191 | root | localhost:53923 | fog | Query | 124 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 192 | root | localhost:53924 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 193 | root | localhost:53925 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 194 | root | localhost:53926 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 195 | root | localhost:53927 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 196 | root | localhost:53928 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 197 | root | localhost:53929 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 198 | root | localhost:53930 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 199 | root | localhost:53932 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 200 | root | localhost:53933 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 201 | root | localhost:53934 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 202 | root | localhost:53935 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 203 | root | localhost:53937 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 204 | root | localhost:53936 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 205 | root | localhost:53939 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 206 | root | localhost:53941 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 207 | root | localhost:53940 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 208 | root | localhost:53943 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 209 | root | localhost:53945 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 210 | root | localhost:53944 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 211 | root | localhost:53947 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 212 | root | localhost:53948 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 213 | root | localhost:53949 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 214 | root | localhost:53950 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 215 | root | localhost:53951 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 216 | root | localhost:53952 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 217 | root | localhost:53953 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 218 | root | localhost:53954 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 219 | root | localhost:53955 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 220 | root | localhost:53956 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 221 | root | localhost:53957 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 222 | root | localhost:53958 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 223 | root | localhost:53959 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 224 | root | localhost:53960 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 225 | root | localhost:53961 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 226 | root | localhost:53962 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 227 | root | localhost:53963 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 228 | root | localhost:53964 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 229 | root | localhost:53965 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 230 | root | localhost:53966 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 231 | root | localhost:53967 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 232 | root | localhost:53968 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 233 | root | localhost:53969 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 234 | root | localhost:53970 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 235 | root | localhost:53971 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 236 | root | localhost:53972 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 237 | root | localhost:53973 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 238 | root | localhost:53974 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 239 | root | localhost:53976 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 240 | root | localhost:53977 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 241 | root | localhost:53978 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 242 | root | localhost:53979 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 243 | root | localhost:53980 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 244 | root | localhost:53981 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 245 | root | localhost:53982 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 246 | root | localhost:53983 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 247 | root | localhost:53984 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 248 | root | localhost:53985 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 249 | root | localhost:53986 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 250 | root | localhost:53987 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 251 | root | localhost:53988 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 252 | root | localhost:53989 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 253 | root | localhost:53990 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 254 | root | localhost:53991 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 255 | root | localhost:53992 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 256 | root | localhost:53993 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 257 | root | localhost:53994 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 258 | root | localhost:53996 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 259 | root | localhost:53997 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 260 | root | localhost:53998 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 261 | root | localhost:53999 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 262 | root | localhost:54000 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 263 | root | localhost:54001 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 264 | root | localhost:54002 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 265 | root | localhost:54003 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 266 | root | localhost:54004 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 267 | root | localhost:54005 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 268 | root | localhost:54006 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 269 | root | localhost:54007 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 270 | root | localhost:54008 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 271 | root | localhost:54009 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 272 | root | localhost:54011 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 273 | root | localhost:54012 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 274 | root | localhost:54014 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 275 | root | localhost:54013 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 276 | root | localhost:54016 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 277 | root | localhost:54017 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 278 | root | localhost:54018 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 279 | root | localhost:54019 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 280 | root | localhost:54020 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 281 | root | localhost:54021 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 282 | root | localhost:54022 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 283 | root | localhost:54023 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 284 | root | localhost:54024 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 285 | root | localhost:54025 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 286 | root | localhost:54026 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 287 | root | localhost:54027 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 288 | root | localhost:54028 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 289 | root | localhost:54029 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 290 | root | localhost:54030 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 291 | root | localhost:54031 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 292 | root | localhost:54032 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 293 | root | localhost:54033 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 294 | root | localhost:54034 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 295 | root | localhost:54035 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 296 | root | localhost:54036 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 297 | root | localhost:54037 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 298 | root | localhost:54038 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 299 | root | localhost:54039 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 300 | root | localhost:54040 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 301 | root | localhost:54042 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 302 | root | localhost:54041 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 303 | root | localhost:54044 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 304 | root | localhost:54045 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 305 | root | localhost:54046 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 306 | root | localhost:54047 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 307 | root | localhost:54048 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 308 | root | localhost:54049 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 309 | root | localhost:54050 | fog | Query | 124 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 310 | root | localhost:54051 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 311 | root | localhost:54052 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 312 | root | localhost:54053 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 313 | root | localhost:54054 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 314 | root | localhost:54055 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 315 | root | localhost:54056 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 316 | root | localhost:54057 | fog | Query | 124 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 317 | root | localhost:54058 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 318 | root | localhost:54059 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 319 | root | localhost:54060 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 320 | root | localhost:54061 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 321 | root | localhost:54062 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 322 | root | localhost:54063 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 323 | root | localhost:54064 | fog | Query | 121 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 324 | root | localhost:54065 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 325 | root | localhost:54067 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 326 | root | localhost:54066 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 327 | root | localhost:54068 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 328 | root | localhost:54072 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 329 | root | localhost:54073 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 330 | root | localhost:54074 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 331 | root | localhost:54075 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 332 | root | localhost:54077 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 333 | root | localhost:54078 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 334 | root | localhost:54079 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 335 | root | localhost:54080 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 336 | root | localhost:54081 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 337 | root | localhost:54082 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 338 | root | localhost:54083 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 339 | root | localhost:54084 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 340 | root | localhost:54086 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 341 | root | localhost:54085 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 342 | root | localhost:54088 | fog | Query | 117 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 343 | root | localhost:54089 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 344 | root | localhost:54090 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 345 | root | localhost:54091 | fog | Query | 116 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 346 | root | localhost:54092 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 347 | root | localhost:54093 | fog | Query | 123 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 348 | root | localhost:54094 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 349 | root | localhost:54095 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 350 | root | localhost:54096 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 351 | root | localhost:54097 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 352 | root | localhost:54098 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 353 | root | localhost:54099 | fog | Query | 118 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 354 | root | localhost:54100 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 355 | root | localhost:54101 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 356 | root | localhost:54102 | fog | Query | 124 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 357 | root | localhost:54103 | fog | Query | 122 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 358 | root | localhost:54104 | fog | Query | 124 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 359 | root | localhost:54105 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 360 | root | localhost:54106 | fog | Query | 119 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 361 | root | localhost:54107 | fog | Query | 122 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 362 | root | localhost:54108 | fog | Query | 120 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 363 | root | localhost:54109 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 364 | root | localhost:54110 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 365 | root | localhost:54111 | fog | Query | 115 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 366 | root | localhost:54112 | fog | Query | 119 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 367 | root | localhost:54113 | fog | Query | 120 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 368 | root | localhost:54114 | fog | Query | 117 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 369 | root | localhost:54115 | fog | Query | 123 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 370 | root | localhost:54116 | fog | Query | 118 | Waiting for global read lock | DELETE FROM aloLog WHERE alID=0 |
| 371 | root | localhost:54117 | fog | Query | 121 | Waiting for global read lock | DELETE FROM multicastSessions WHERE msID=0 OR msNFSGroupID=0 |
| 372 | root | localhost:54119 | fog | Sleep | 19 | | NULL |
| 373 | root | localhost:54120 | fog | Sleep | 6 | | NULL |
| 374 | root | localhost:54121 | fog | Sleep | 19 | | NULL |
| 375 | root | localhost:54122 | fog | Sleep | 19 | | NULL |
| 376 | root | localhost:54123 | fog | Sleep | 19 | | NULL |
| 380 | root | localhost | fog | Query | 0 | NULL | show full processlist |
±----±-----±----------------±-----±--------±-----±-----------------------------±-------------------------------------------------------------+
206 rows in set (0.00 sec)

Tom Elliott

Removed that method and caller so maybe now it’ll be better.

Joseph Hales

The web page seems to be working now but I am still having some issues and the Apache stops logging a min after restart with the same errors as shown in the attached log.
Apache.log.txt

Wayne Workman

@Joseph-Hales I wonder if the sheer amount of errors being recorded simply crash the logging mechanism… ?

There are levels of logging in daemons, you can adjust the level on a scale, some have different modes.

This was written for Ubuntu, but I would imagine that the things in here can be applied to any distro very similarly:
https://www.digitalocean.com/community/tutorials/how-to-configure-logging-and-log-rotation-in-apache-on-an-ubuntu-vps

It might be beneficial to turn up the verbosity of logging just to figure out what’s going on… and it might be beneficial to turn the verbosity down to lower cpu load … and so logging doesn’t crash.

Tom Elliott

I’m not 100% sure how much I like approaching things in this manner, but i’ve now made it so FOG will die if the db connection cannot happen. This should stop spamming logs with “mysqli::query() couldn’t fetch mysqli” error messages. It also “deadens” the page right then and there. This way resources aren’t trying to get data as in the “main page” as used to be the case.

This isn’t going to fix the issue here, but should help in lessening errors being thrown all over the place.

Wayne Workman

@Tom-Elliott said:

i’ve now made it so FOG will die if the db connection cannot happen.

Can you elaborate on that? Will it try to come alive later on? If the DB connection is broke, can fog try to restart mysql? Can there be a limit on how often it gets restarted? Sounds like it’s living dangerously.

Joseph Hales

As of right now fog has been running for 18 hours when I run

show full processlist;

I get 191 lines of

| 43572 | root | localhost:35810 | fog | Sleep | 3 | | NULL

at least I’m not seeing the locked delete entries any more. I think I have a old deleted snapin associated with some hosts what would be a command to select and delete it from my sql it may be causing the apache errors I am getting now.

Apache.log.txt

Wayne Workman

@Joseph-Hales Not sure… I’d have to have my hands on it to build the right command for you.

Just work on building a select command. Once you can get the data you want gone selected, then just convert it to a delete command.

Tom Elliott

Is it safe to solve this thread now? The process list isn’t bad by itself as you’re most likely using persistent connections. Many of those processes are sleeping because of the persistence.

Joseph Hales

I’m worried about the sigterm termination and the max hosts error every time MySQL starts I will be switching hardware soon and if these are underlying database issues I would like to resolve them before migration.

Joseph Hales

moving this to solved as we are now on a much higher SVN and about to replace the server entirely.

Tom Elliott

@Joseph-Hales I agree with the plan to solve the thread but not for the reasons you describe, rather for the fact this was likely somewhat related to the issues found earlier.

After Update to SVN 4928 web pages time out constantly

158

12.0k

17.3k

155.2k