Fog web interface times out loading almost any page

[Joseph Hales]

Ubuntu 14 SVN 4602 web pages fail to load at all unless retried several times this is no matter which icon is clicked the apache error log shows this error repeating.

[Fri Sep 11 13:08:44.997022 2015] [:error] [pid 9764] [client 10.109.49.75:51920] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 64
[Fri Sep 11 13:08:44.997054 2015] [:error] [pid 9764] [client 10.109.49.75:51920] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 64
[Fri Sep 11 13:08:44.997101 2015] [:error] [pid 9764] [client 10.109.49.75:51920] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 64
[Fri Sep 11 13:08:44.997135 2015] [:error] [pid 9764] [client 10.109.49.75:51920] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 64
[Fri Sep 11 13:08:44.997167 2015] [:error] [pid 9764] [client 10.109.49.75:51920] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 64
[Fri Sep 11 13:08:44.997199 2015] [:error] [pid 9764] [client 10.109.49.75:51920] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 64
[Fri Sep 11 13:08:44.997230 2015] [:error] [pid 9764] [client 10.109.49.75:51920] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 64
[Fri Sep 11 13:08:44.997262 2015] [:error] [pid 9764] [client 10.109.49.75

[Tom Elliott]

Whats your system disk space at? Is it getting close to max? It sounds like mysql is losing room, then gaining just enough to kind of keep running.

[Joseph Hales]

disk space is fine 50% used on the terabyte /images and /root is on using 19% of a 250 gb drive.

[Joseph Hales]

it does seem to go away if my active users is low typically when the issues occurs i have 5 or more users logged in.

[Wayne Workman]

@Joseph-Hales What sort of processing power does your fog server have?

Can you try to monitor the CPU usage via “top” ?

[Joseph Hales]

Here is what top looks like.
top - 10:45:03 up 3 days, 2:16, 1 user, load average: 10.50, 8.93, 7.66
Tasks: 470 total, 3 running, 467 sleeping, 0 stopped, 0 zombie
%Cpu(s): 21.1 us, 2.6 sy, 0.0 ni, 74.2 id, 1.5 wa, 0.0 hi, 0.6 si, 0.0 st
KiB Mem: 24678932 total, 24463228 used, 215704 free, 998420 buffers
KiB Swap: 4844540 total, 0 used, 4844540 free. 20614724 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13693 mysql 20 0 8722412 351992 7352 S 163.1 1.4 4024:15 mysqld
7375 www-data 20 0 564848 255004 7280 R 89.2 1.0 0:55.44 apache2
7659 www-data 20 0 327004 16804 5728 S 6.0 0.1 0:32.13 apache2
1617 www-data 20 0 326748 17032 6084 S 5.3 0.1 1:22.77 apache2
7637 www-data 20 0 326524 16484 5732 S 3.0 0.1 0:31.86 apache2
5722 www-data 20 0 327076 17464 6112 S 2.7 0.1 0:54.97 apache2
1659 www-data 20 0 327036 19084 7796 S 2.3 0.1 1:38.26 apache2
5847 www-data 20 0 327004 17320 6028 S 2.3 0.1 0:53.62 apache2
7322 www-data 20 0 327036 17148 5988 S 2.3 0.1 0:36.12 apache2
7445 www-data 20 0 326748 17128 6048 S 2.3 0.1 0:38.99 apache2
7451 www-data 20 0 327004 17220 6020 S 2.3 0.1 0:34.73 apache2
7527 www-data 20 0 326748 16972 6008 S 2.3 0.1 0:33.10 apache2
7550 www-data 20 0 327168 17532 6000 S 2.3 0.1 0:34.11 apache2
7607 www-data 20 0 326972 17036 5984 S 2.3 0.1 0:32.40 apache2
7611 www-data 20 0 326748 16988 5976 S 2.3 0.1 0:34.47 apache2
7711 www-data 20 0 476540 20000 8728 S 2.3 0.1 0:32.78 apache2
7713 www-data 20 0 326972 17152 5988 S 2.3 0.1 0:32.41 apache2
1262 www-data 20 0 327004 17472 6144 S 2.0 0.1 1:34.06 apache2
1691 www-data 20 0 326672 18168 7260 S 2.0 0.1 1:26.96 apache2
1950 www-data 20 0 326748 17064 6080 S 2.0 0.1 1:22.28 apache2
5724 www-data 20 0 326972 17364 6064 S 2.0 0.1 0:54.74 apache2
5771 www-data 20 0 326492 16724 6036 S 2.0 0.1 0:54.01 apache2
5851 www-data 20 0 327216 17512 6032 S 2.0 0.1 0:53.55 apache2
7345 www-data 20 0 327004 17096 6004 S 2.0 0.1 0:35.27 apache2
7374 www-data 20 0 326972 17148 6052 S 2.0 0.1 0:33.98 apache2
7415 www-data 20 0 327004 17312 6008 S 2.0 0.1 0:35.58 apache2
7431 www-data 20 0 326972 17104 5992 S 2.0 0.1 0:33.74 apache2
7433 www-data 20 0 326972 17260 5992 S 2.0 0.1 0:36.23 apache2
7492 www-data 20 0 326748 16852 5824 S 2.0 0.1 0:33.44 apache2
7511 www-data 20 0 327004 17188 6008 S 2.0 0.1 0:32.97 apache2
7524 www-data 20 0 326716 16972 5996 S 2.0 0.1 0:33.87 apache2
7564 www-data 20 0 326460 16776 5984 S 2.0 0.1 0:33.87 apache2
7576 www-data 20 0 326716 16864 5980 S 2.0 0.1 0:32.53 apache2
7582 www-data 20 0 326492 16832 6008 S 2.0 0.1 0:33.65 apache2
7592 www-data 20 0 327004 18208 7124 S 2.0 0.1 0:31.81 apache2
7617 www-data 20 0 326972 17296 6008 S 2.0 0.1 0:33.30 apache2
7649 www-data 20 0 326972 17296 5996 S 2.0 0.1 0:32.44 apache2

[Wayne Workman]

@Joseph-Hales That’s a lot of Apache2 there, Joe…

How many clients do you have running the fog client? what’s your check-in time set to?

[Joseph Hales]

FOG_SERVICE_CHECKIN_TIME 60
Hosts found 6781
I updated to SVN 4666 this morning I don’t see the failure to load issue so far but not many users are logged in yet but page load times are about 5 seconds on the host search page when searching for one host and listing all hosts takes about 50 seconds I also get the install database upgrade page occasionally at random times.

[Joseph Hales]

I also am only running the legacy client at this time. Finally the home page is always displaying that there are pending MAC’s that need approval but when I go to approve it fails to find any.

[Wayne Workman]

@Joseph-Hales said:

I also am only running the legacy client at this time. Finally the home page is always displaying that there are pending MAC’s that need approval but when I go to approve it fails to find any.

Is your setup virtualized and do you have a snapshot ? Do you have DB backups available ?

For example, I take a snapshot before every update, and export my DB stuff before every update. I keep the last two snapshots and ALL the DB backups. Plus I use samba to easily backup my images too.

I only ask about the backups because of this https://forums.fogproject.org/topic/5716/some-clients-disappear-from-web-interface-but-are-still-present-in-databse

[Joseph Hales]

No its not virtualized and if I loose some client records its not the end of the world I may be seeing this issues as well if the mac address’s are deleted should I clean out the database of these records. If so can you give me the mysql command to do this

[Wayne Workman]

@Joseph-Hales said:

No its not virtualized and if I loose some client records its not the end of the world I may be seeing this issues as well if the mac address’s are deleted should I clean out the database of these records. If so can you give me the mysql command to do this

I’m 100% sure we can give you a command to do that, but if you have a host export you can just re import the data.

I have not tested this so be forewarned… backup your DB before hand.
FOG Configuration -> Configuration Save -> Export

DELETE FROM hosts WHERE hostID NOT IN (SELECT hmhostID FROM hostMAC);

[Joseph Hales]

I thought that command just saved the fog settings not the hosts and images?

[Joseph Hales]

When I try to connect to mysql it errors out with error too many connections I will update svn, reboot and try again.

[Joseph Hales]

Still run into the error ERROR 1040 (HY000): Too many connections

[Wayne Workman]

@Joseph-Hales said:

Still run into the error ERROR 1040 (HY000): Too many connections

I know what your issue is… you have tons of hosts… you need to increase your maximum connections!!!

There’s a section in this about just that https://wiki.fogproject.org/wiki/index.php/Troubleshoot_MySQL

[Wayne Workman]

@Joseph-Hales said:

I thought that command just saved the fog settings not the hosts and images?

That feature exports the entire DB. Everything.

[Joseph Hales]

Ok if someone could make a note in the wiki the my.cnf in ubuntu 14 seems to be located in /etc/mysql/ not the root of /etc/ that fixed my too many connections. It seems the update to 14 deleted my old my.cnf and reverted to the default 100 max connections. after clearing the bad records it seems I lost 1050 records. If I import an older backup made from the web page will it only add records that no longer exist or will it overwrite and delete the new records added since the backup?

[Joseph Hales]

Ok pages seem better and server load is down but my Task Management page fails to load at all, every other menu option loads normally.

[Wayne Workman]

@Joseph-Hales said:

If I import an older backup made from the web page will it only add records that no longer exist or will it overwrite and delete the new records added since the backup?

An import will blast your DB and place the import in place. You will loose all changes made since that backup was made.
Of course this doesn’t affect the ACTUAL images made since then, it just affects the image definitions in the DB. Those can be recreated manually via the Web UI. But you have to create them exactly as they were (image type, OS, Exact path and the path is case sensitive).

And I’ll make note in the WiKi about Ubuntu 14.