MYSQL connections max out and slow down fog server
-
I’ve been having issues with my fog server since I updated it to 1.2. I was on .32 and did a fresh install and imported my fog.sql database in. It worked for a few days just fine but now it is getting hammered with open connections to the sql database. It’ll hit 150 in no time and that causes my CPU to max out at 100% and the fog server becomes nearly unusable. I’m on a Dell Optiplex 380. Its worked just fine for ages now and nothing on my school site has changed. It has 2 gigs of ram and is running ubuntu 14.04 LTS.
If I restart the mysql service everything immediately clears up, the CPU goes back to near idle and everything is smooth but as soon as mysql starts back up the connections flood in and its back to running like a brick.
I am a linux novice but have been running FOG for about 8 years (version .10 if i recall correctly). I’m about to go to lunch so if I don’t reply quickly, plz forgive me.
i use conky to monitor a few things for quick reference and thats where I see all the connections and my cpu usage maxed. I installed iftop to monitor connections and its just a constant list of the computers connecting to the server.
-
I’d recommend, if you’re using 1.2.0, sets your database configuration to persistent:
In /var/www/fog/lib/fog/Config.class.php edit the DATABASE_HOST line from:
[php]define(‘DATABASE_HOST’,‘localhost’);[/php] to [php]define(‘DATABASE_HOST’,‘p:127.0.0.1’); [/php]and all should normalize a little better.
-
If your running ubuntu the issue can be php session purging fix instructions here [url]http://www.fogproject.org/wiki/index.php/Fog_Tweaks[/url]
-
I tried the persistent DB option earlier i found on a different thread and that didn’t work. Lokoing at the php session purging fix, this is what my line looks like:
09,39 * * * * root [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime)
It doesn’t look like the fog wiki page. Should I still change it to match?
-
Only if your running ubuntu its an os specific fix I believe.
-
I am on Ubuntu but i figured the before line would match. it doesn’t.
-
Its the same on my copy of ubuntu 12.04 64bit server what version are you running?
-
14.04 64bit (mentioned in OP)
-
Comment it out so you can switch it back but I know its fixed simular behavior on mine. Check the wiki I know there are issues with 14 and that its not currently recomended. [url]http://www.fogproject.org/wiki/index.php/Ubuntu_14.04[/url]
-
I’ll try it now. I know the TFTP issue exists but after I start the service manually, all seems fine.
[[EDIT]]
It was running smoother for several minutes but both CPU cores are at 100% again, inbound connections is up to 250 now with outbound at 150 (outbound was always at 1 before). The web gui is back to being unusable.
I was thinking of going to CentOS. If this issue is specific to Ubuntu, maybe its time to get rid of Ubuntu.
-
Perhaps but I have good luck with 12.04 lts and its easier for me to find cut and paste solutions equal to my noob skills.
-
Ok, I got CentOS6.5 running, got FOG installed and imported my fog.sql database. The issue returns. 100% CPU usage and become nearly unusable. I have to ask the question I really dont’ want to ask. Is it my imported fog.sql database. I’ve used the same database for years and has nearly a 1000 host entries, over 200 printers configured for installation, over 100 snapin entries applied to countless computers. I really REALLY don’t want to have to recreate my database, that would be a massive amount of work but could my database be the cause? If I stop the mysql service, the computer goes back to normal.
-
I don’t think it’s the imported database.
I’m going on a limb and when you did the install, did you upgrade the system or fresh install? When we first upgraded we saw a similar issue, and all I had to do to correct the issue was allow for more hosts to connect. I think the default was 100, and we set ours to 250.
-
This is a fresh CentOS install and a fresh fog 1.2 install. Nothing upgraded. On Ubuntu, it was a fresh install of both as well. I’ve made sure too that all passwords are the same as well.
-
Check your inbox, I’d rather talk over Google Hangouts or GTALK if you’re able to and sent a message to you there containing my account stuff.
-
Just remember to post back here if you fix it so we can see the solution.
-
We fixed the issue.
As the db was imported, the old snapinTasks and snapinJobs tables were still there. THis was causing the hosts to checkin and repeatedly spamming for the data that was not relevant.
Running:
[code]truncate table snapinJobs;truncate table snapinTasks;[/code] fixed the problem.
-
[quote=“Joseph Hales, post: 43691, member: 18131”]Just remember to post back here if you fix it so we can see the solution. ;-)[/quote]
Tom is going to post the results. it did get solved. I can’t really go into the fix as it is way above my level of knowledge but he said eh was going to post here for others to see.
A big thanks to him for the help he provided.
-
I almost think we should make that truncate part of the upgrade script. Except it screws with the logs doesn’t it?
-
[quote=“Joseph Hales, post: 43723, member: 18131”]I almost think we should make that truncate part of the upgrade script.[/quote]
Also part of the backup script. I backup my database every 2 weeks.