After Update to SVN 4928 web pages time out constantly

[Joseph Hales]

ubuntu 14 svn 1428 seems to timeout on webpages when accesing fog webmin is unaffected the last few lines of the apache log are as follows.

[Mon Oct 12 14:45:41.863372 2015] [:error] [pid 5189] [client 10.6.49.35:58113] PHP Warning: session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/User.class.php on line 112
[Mon Oct 12 14:45:41.943332 2015] [:error] [pid 5394] [client 10.109.49.1:49561] PHP Warning: session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/User.class.php on line 112
[Mon Oct 12 14:45:42.017669 2015] [:error] [pid 5337] [client 10.114.51.24:59299] PHP Warning: session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/User.class.php on line 112
[Mon Oct 12 14:45:42.097035 2015] [:error] [pid 5336] [client 10.109.49.9:59779] PHP Warning: session_destroy(): Trying to destroy uninitialized session in /var/www/html/fog/lib/fog/User.class.php on line 112

[Joseph Hales]

I updated SVN again this morning still unable to access the web interface now my apache log shows the following.

[Tue Oct 13 08:16:48.299162 2015] [:error] [pid 13666] [client 10.110.51.17:54649] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 08:16:48.302077 2015] [:error] [pid 13666] [client 10.110.51.17:54649] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 08:16:48.302427 2015] [:error] [pid 13666] [client 10.110.51.17:54649] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 08:16:48.302755 2015] [:error] [pid 13666] [client 10.110.51.17:54649] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 08:16:48.676852 2015] [mpm_prefork:notice] [pid 13555] AH00169: caught SIGTERM, shutting down
[Tue Oct 13 08:19:08.643031 2015] [mpm_prefork:notice] [pid 1502] AH00163: Apache/2.4.16 (Ubuntu) OpenSSL/1.0.1f configured – resuming normal operations
[Tue Oct 13 08:19:08.704182 2015] [core:notice] [pid 1502] AH00094: Command line: ‘/usr/sbin/apache2’
[Tue Oct 13 08:19:50.925228 2015] [mpm_prefork:error] [pid 1502] AH00161: server reached MaxRequestWorkers setting, consider raisin

[Joseph Hales]

I have already doubled the MaxRequestWorkers to 300 from 150 and restarted the server. same result.

[Joseph Hales]

Checked out revision 4146

[Joseph Hales]

I increased max connections in mysql and am getting some responses from the web interface but only partial and extremly slow. I am still getting a MaxRequestWorkers error so I will try increasing that some more but I think there is an underlying issue I am just kludging around by throwing resources at it.

[Joseph Hales]

The web interface is still down and the following is the latest apache log entries MaxWorkers is at 500 now.

[Tue Oct 13 09:20:05.735019 2015] [:error] [pid 9950] [client 10.109.48.112:60149] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 09:20:05.735359 2015] [:error] [pid 9950] [client 10.109.48.112:60149] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 09:20:05.735730 2015] [:error] [pid 9950] [client 10.109.48.112:60149] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 09:20:05.736063 2015] [:error] [pid 9950] [client 10.109.48.112:60149] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 09:20:05.738972 2015] [:error] [pid 9950] [client 10.109.48.112:60149] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 09:20:05.739325 2015] [:error] [pid 9950] [client 10.109.48.112:60149] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 09:20:05.739662 2015] [:error] [pid 9950] [client 10.109.48.112:60149] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 09:20:05.900031 2015] [:error] [pid 9880] [client 10.41.50.221:49939] PHP Warning: mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/db/MySQL.class.php on line 35
[Tue Oct 13 09:20:05.900146 2015] [:error] [pid 9880] [client 10.41.50.221:49939] PHP Warning: mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/db/MySQL.class.php on line 37
[Tue Oct 13 09:20:05.900245 2015] [:error] [pid 9880] [client 10.41.50.221:49939] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 09:20:09.267095 2015] [core:warn] [pid 9738] AH00045: child process 9843 still did not exit, sending a SIGTERM
[Tue Oct 13 09:20:11.269473 2015] [core:warn] [pid 9738] AH00045: child process 9843 still did not exit, sending a SIGTERM
[Tue Oct 13 09:20:13.271827 2015] [core:warn] [pid 9738] AH00045: child process 9843 still did not exit, sending a SIGTERM
[Tue Oct 13 09:20:15.274178 2015] [core:error] [pid 9738] AH00046: child process 9843 still did not exit, sending a SIGKILL
[Tue Oct 13 09:20:16.275460 2015] [mpm_prefork:notice] [pid 9738] AH00169: caught SIGTERM, shutting down
[Tue Oct 13 09:22:34.694733 2015] [mpm_prefork:notice] [pid 1518] AH00163: Apache/2.4.16 (Ubuntu) OpenSSL/1.0.1f configured – resuming normal operations
[Tue Oct 13 09:22:34.736477 2015] [core:notice] [pid 1518] AH00094: Command line: ‘/usr/sbin/apache2’
[Tue Oct 13 09:23:17.953521 2015] [mpm_prefork:error] [pid 1518] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

[Wayne Workman]

@Joseph-Hales Have you rebooted the fog server? Try to restart MySQL? Restart httpd (apache) ?

[Joseph Hales]

Yes every time I have changed one of the settings in apache or mysql I have first restarted both services and if it still fails I reboot the entire server same result.

[Wayne Workman]

@Joseph-Hales Try to turn down your check-in time even further?

[Joseph Hales]

Where can I change that outside of the web interface. Also this just cropped up suddenly when I updated SVN from 4700 to 4928 and up no other changes the only reason I updated SVN at all at this point is to fix a known issue with new and updated snapins.

[Wayne Workman]

@Joseph-Hales
Ignore the second circle, that’s for reboot/shutdown wait time.

[Joseph Hales]

The web interface is not responding that is the whole issue so I am unable to access that setting.

[Wayne Workman]

@Joseph-Hales Oh duh I’m sorry. Do it via CLI then.

mysql
use fog
update globalSettings set settingValue = 300 where settingKey = 'FOG_Service_Checkin_Time';

[Joseph Hales]

@Wayne-Workman said:

update globalSettings set settingValue = 300 where settingKey = ‘FOG_Service_Checkin_Time’;

Thank you now my web interface is responding did a recent SVN reset this value?

[Wayne Workman]

@Joseph-Hales I have no idea, but I seriously doubt the @Developers would write any code that would intentionally change your settings.

Is it possible that the installer updates this field to the default I wonder?

[Joseph Hales]

I spoke too soon after a few moments it stopped responding but I am not seeing any new errors in the apache log this time. I will try restarting the server itself instead of just the service.

[Joseph Hales]

No the web server is still not responding and I am back to getting the same apache errors.

[Tue Oct 13 10:58:17.303264 2015] [:error] [pid 7864] [client 10.1.51.37:64087] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 10:58:17.303784 2015] [:error] [pid 7864] [client 10.1.51.37:64087] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 10:58:17.304310 2015] [:error] [pid 7864] [client 10.1.51.37:64087] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 10:58:17.304799 2015] [:error] [pid 7864] [client 10.1.51.37:64087] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 10:58:17.305287 2015] [:error] [pid 7864] [client 10.1.51.37:64087] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 10:58:17.305771 2015] [:error] [pid 7864] [client 10.1.51.37:64087] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 10:58:17.306251 2015] [:error] [pid 7864] [client 10.1.51.37:64087] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 62
[Tue Oct 13 10:58:17.661588 2015] [mpm_prefork:notice] [pid 1518] AH00169: caught SIGTERM, shutting down
[Tue Oct 13 11:00:51.298115 2015] [mpm_prefork:notice] [pid 1544] AH00163: Apache/2.4.16 (Ubuntu) OpenSSL/1.0.1f configured – resuming normal operations
[Tue Oct 13 11:00:51.348300 2015] [core:notice] [pid 1544] AH00094: Command line: ‘/usr/sbin/apache2’
[Tue Oct 13 11:01:34.564492 2015] [mpm_prefork:error] [pid 1544] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

[Wayne Workman]

@Joseph-Hales what does the command top show ?

[Joseph Hales]

top - 11:19:05 up 18 min,  1 user,  load average: 2.83, 2.84, 2.19
Tasks: 463 total,   1 running, 462 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.1 us,  0.8 sy,  0.0 ni, 84.6 id,  4.2 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem:  24678932 total,  2849080 used, 21829852 free,   134548 buffers
KiB Swap:  4844540 total,        0 used,  4844540 free.  1368608 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1419 mysql     20   0 8721920 193320   7612 S 139.0  0.8  24:30.19 mysqld
 2647 www-data  20   0  325384  14092   5248 S   9.6  0.1   0:03.52 apache2
 2642 www-data  20   0  325628  14776   5684 S   1.7  0.1   0:02.77 apache2
 3452 www-data  20   0  325628  14324   5228 S   1.7  0.1   0:02.56 apache2
 2653 www-data  20   0  325628  14296   5212 S   1.3  0.1   0:02.58 apache2
 2975 www-data  20   0  325376  14880   6040 S   1.3  0.1   0:03.06 apache2
 2982 www-data  20   0  325628  14308   5224 S   1.3  0.1   0:03.38 apache2
 2993 www-data  20   0  325628  14324   5236 S   1.3  0.1   0:03.42 apache2
 3002 www-data  20   0  325628  14332   5244 S   1.3  0.1   0:03.14 apache2
 3439 www-data  20   0  325628  14296   5212 S   1.3  0.1   0:03.72 apache2
 3520 www-data  20   0  324092  13088   5532 S   1.3  0.1   0:02.69 apache2
 3523 www-data  20   0  325372  14092   5252 S   1.3  0.1   0:03.21 apache2
 3524 www-data  20   0  324348  13056   5252 S   1.3  0.1   0:03.24 apache2
 3532 www-data  20   0  325628  14632   5536 S   1.3  0.1   0:03.17 apache2
 3699 www-data  20   0  325628  14316   5232 S   1.3  0.1   0:04.04 apache2
 3764 www-data  20   0  325628  14352   5268 S   1.3  0.1   0:02.65 apache2
 3459 www-data  20   0  325628  14368   5284 S   1.0  0.1   0:02.37 apache2

Mod edited to use code box.

[Wayne Workman]

@Joseph-Hales How many cores does this server have? How much RAM? What’s the core speed? Is the storage SSD or mechanical?

I’m not sure how it’s possible for MySQL to use 139% of a CPU…