RC10 Broken Items on upgrade

[adukes40]

@Sebastian-Roth I just learned how to tail a file and dump it to a file last night at class lol. IT WORKED. (sorry small linux golfclap for myself) Anyway here is the last 25 lines of the error.log

[Tue Sep 13 07:42:16.555758 2016] [:error] [pid 15621] [client 10.103.69.25:50786] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 899
[Tue Sep 13 07:42:16.594268 2016] [:error] [pid 15532] [client 10.103.12.229:62782] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Sep 13 07:42:17.556278 2016] [:error] [pid 15742] [client 10.103.65.30:50729] PHP Warning: PDO::query(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 196
[Tue Sep 13 07:42:17.556331 2016] [:error] [pid 15742] [client 10.103.65.30:50729] PHP Warning: PDO::query(): Error reading result set’s header in /var/www/html/fog/lib/db/pdodb.class.php on line 196
[Tue Sep 13 07:42:17.556427 2016] [:error] [pid 15742] [client 10.103.65.30:50729] PHP Strict Standards: Non-static method FOGBase::debug() should not be called statically in /var/www/html/fog/lib/db/pdodb.class.php on line 208
[Tue Sep 13 07:42:17.897870 2016] [:error] [pid 15472] [client 10.104.12.57:50771] PHP Warning: PDO::query(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 196
[Tue Sep 13 07:42:17.897931 2016] [:error] [pid 15472] [client 10.104.12.57:50771] PHP Warning: PDO::query(): Error reading result set’s header in /var/www/html/fog/lib/db/pdodb.class.php on line 196
[Tue Sep 13 07:42:17.898069 2016] [:error] [pid 15472] [client 10.104.12.57:50771] PHP Strict Standards: Non-static method FOGBase::debug() should not be called statically in /var/www/html/fog/lib/db/pdodb.class.php on line 208
[Tue Sep 13 07:42:17.940687 2016] [:error] [pid 15710] [client 10.103.66.15:50724] PHP Warning: PDO::query(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 196
[Tue Sep 13 07:42:17.940745 2016] [:error] [pid 15710] [client 10.103.66.15:50724] PHP Warning: PDO::query(): Error reading result set’s header in /var/www/html/fog/lib/db/pdodb.class.php on line 196
[Tue Sep 13 07:42:17.940827 2016] [:error] [pid 15710] [client 10.103.66.15:50724] PHP Strict Standards: Non-static method FOGBase::debug() should not be called statically in /var/www/html/fog/lib/db/pdodb.class.php on line 208
[Tue Sep 13 07:42:18.074087 2016] [:error] [pid 15671] [client 10.103.13.134:50659] PHP Warning: PDO::query(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 196
[Tue Sep 13 07:42:18.074205 2016] [:error] [pid 15671] [client 10.103.13.134:50659] PHP Warning: PDO::query(): Error reading result set’s header in /var/www/html/fog/lib/db/pdodb.class.php on line 196
[Tue Sep 13 07:42:18.074304 2016] [:error] [pid 15671] [client 10.103.13.134:50659] PHP Strict Standards: Non-static method FOGBase::debug() should not be called statically in /var/www/html/fog/lib/db/pdodb.class.php on line 208
[Tue Sep 13 07:42:19.016080 2016] [:error] [pid 15558] [client 10.105.65.97:61732] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Sep 13 07:42:19.312576 2016] [:error] [pid 15577] [client 10.103.67.98:50743] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Sep 13 07:42:19.724552 2016] [:error] [pid 15773] [client 10.102.12.235:50724] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Sep 13 07:42:19.824479 2016] [:error] [pid 15667] [client 10.103.14.42:50786] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
[Tue Sep 13 07:42:21.895462 2016] [mpm_prefork:notice] [pid 15466] AH00169: caught SIGTERM, shutting down
[Tue Sep 13 07:42:41.586125 2016] [mpm_prefork:notice] [pid 1098] AH00163: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f configured – resuming normal operations
[Tue Sep 13 07:42:41.587691 2016] [core:notice] [pid 1098] AH00094: Command line: ‘/usr/sbin/apache2’
[Tue Sep 13 07:42:55.604072 2016] [mpm_prefork:error] [pid 1098] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Tue Sep 13 08:47:15.270770 2016] [mpm_prefork:notice] [pid 1105] AH00163: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f configured – resuming normal operations
[Tue Sep 13 08:47:15.359753 2016] [core:notice] [pid 1105] AH00094: Command line: ‘/usr/sbin/apache2’
[Tue Sep 13 08:47:31.006640 2016] [mpm_prefork:error] [pid 1105] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

[adukes40]

On a side note, It seems the remote sites cant get past network boot. If I try to boot a machine from the local site where the master server is, it will work, and get to the FOG pxe screen, but its does take longer than normal.

[Tom Elliott]

@adukes40 can you increase your MySQL max_connections? I’m also going to remove persistent connections from the main and addressed the error of calling method debug statically when it is not a static method. I doubt any one of these things will fix what you’re seeing but based on the information I think your MySQL is just not responding to some connections. This makes php wait which would also explain the Max workers issue you’re seeing. You can try running the rc-11 build to see if it helps out of the box. For this I leave someone else to direct as explaining is a bit hard over a 'mobile phone.

[adukes40]

Ok, so I can just SVN up? and Where would I increase max_connections, and by how much?

[Tom Elliott]

@adukes40 The RC working is done purely through git as branching is greatly simplified. So svn up will not work in your case.

I dont know the os fog is running on but it normally is found in /etc/my.cnf or /etc/mysql/my.cnf but your location might vary.

I’d say start by setting max_connections to 300. Once you make the change you will have to restart the MySQL service.

[Wayne Workman]

@adukes40, I only know how with git. Message me and I’ll give you the commands.

[adukes40]

@Tom-Elliott Before I read this I found the my.cnf file. It currently shows:

#max_connections = 100

Do I also need to remove the # sign?

[Tom Elliott]

@adukes40 yes the Linux class you’re taking probably hasn’t told you yet, typically # in scripting means a comment so anything after that point will not be executed for that line.

[adukes40]

@Tom-Elliott Ok, so max_connections is now set to 300
when I type mysql it lets me in now. it was saying too many connections before. However the GUI is still the same. can’t load/slow. I haven’t updated yet though.

[adukes40]

BTW its almost feels like the slowness when one of the remote nodes can’t contact the master, and causes it to load slower. except in this situation when I see the dashboard, it cannot see any of the remote sites. It shows then on the graph, but this is all I see.

[Tom Elliott]

@adukes40 to test your theory just disable graph enabled on all nodes. For now it’s likely faster to do this than to try testing potentially many other unknowns. Of course this could all be fixed in RC 11 already but I doubt it. There is minimal reliance on nodes being available or not though. It really only needs “access” to see if the node is available or not. This shouldn’t hurt anything but it may delay load up of dashboard page by two seconds per node that’s disabled. This is because I have to check twice if the page is available to generate my URL lists for bandwidth and client checks. I’ll rework it tonight to generate the list of URLs only once on initial load up of the page. This should help limit redundant checks and improve overall availability

If you can

[adukes40]

@Tom-Elliott It is brutal, but I will try to disable the nodes first.Good news is that imaging actually works here at the master site. Takes a bit longer to get to the image part, but its actually makes it. Just anytime it needs to communicate to the DB it takes its sweet time.

[adukes40]

OK, all disabled except the master, but no change in performance.

[Wayne Workman]

@adukes40 graph disabled and node disabled are two different things. Graph disabled just doesn’t report disk usage or bandwidth usage. The node still operates fully.

[adukes40]

@Wayne-Workman All graphs are now disabled. Still having issues connecting to home page. and still looks like this:

[Wayne Workman]

@adukes40 check your cpu usage on your desktop when accessing the web interface. See how much power the browser is using.

Also, the output from top on the server when trying to access the dashboard would be useful.

[adukes40]

@Wayne-Workman Computer CPU - Chrome hit 02% max…

here is best I could get you from TOP

[Wayne Workman]

@adukes40 apache is working pretty hard. And I’m not positive but it looks like some of those apache processes have been running a long time. @Tom-Elliott @george1421?

[Tom Elliott]

@adukes40 said in RC10 Broken Items on upgrade:

Can you reset all host encryption data? Do this from mysql cli

Connect by:
mysql -u root fog

Run command
update hosts set hostSecToc='';

[adukes40]

@Tom-Elliott root@MSDCATS09:~# mysql -u fog
ERROR 1045 (28000): Access denied for user ‘fog’@‘localhost’ (using password: NO)