RC10 Broken Items on upgrade



  • @adukes40 apache is working pretty hard. And I’m not positive but it looks like some of those apache processes have been running a long time. @Tom-Elliott @george1421?



  • @Wayne-Workman Computer CPU - Chrome hit 02% max…

    here is best I could get you from TOP

    0_1473779649323_upload-d8145ae5-beb9-4cfc-9dac-4ab7f22b9e74



  • @adukes40 check your cpu usage on your desktop when accessing the web interface. See how much power the browser is using.

    Also, the output from top on the server when trying to access the dashboard would be useful.



  • @Wayne-Workman All graphs are now disabled. Still having issues connecting to home page. and still looks like this:
    0_1473778655093_upload-34313270-dee1-4467-88af-a386837090c4



  • @adukes40 graph disabled and node disabled are two different things. Graph disabled just doesn’t report disk usage or bandwidth usage. The node still operates fully.



  • OK, all disabled except the master, but no change in performance.



  • @Tom-Elliott It is brutal, but I will try to disable the nodes first.Good news is that imaging actually works here at the master site. Takes a bit longer to get to the image part, but its actually makes it. Just anytime it needs to communicate to the DB it takes its sweet time.


  • Senior Developer

    @adukes40 to test your theory just disable graph enabled on all nodes. For now it’s likely faster to do this than to try testing potentially many other unknowns. Of course this could all be fixed in RC 11 already but I doubt it. There is minimal reliance on nodes being available or not though. It really only needs “access” to see if the node is available or not. This shouldn’t hurt anything but it may delay load up of dashboard page by two seconds per node that’s disabled. This is because I have to check twice if the page is available to generate my URL lists for bandwidth and client checks. I’ll rework it tonight to generate the list of URLs only once on initial load up of the page. This should help limit redundant checks and improve overall availability

    If you can



  • BTW its almost feels like the slowness when one of the remote nodes can’t contact the master, and causes it to load slower. except in this situation when I see the dashboard, it cannot see any of the remote sites. It shows then on the graph, but this is all I see.

    0_1473775084148_upload-179956b5-f99f-476d-b1f7-5f953f2e9284



  • @Tom-Elliott Ok, so max_connections is now set to 300
    when I type mysql it lets me in now. it was saying too many connections before. However the GUI is still the same. can’t load/slow. I haven’t updated yet though.


  • Senior Developer

    @adukes40 yes the Linux class you’re taking probably hasn’t told you yet, typically # in scripting means a comment so anything after that point will not be executed for that line.



  • @Tom-Elliott Before I read this I found the my.cnf file. It currently shows:

    #max_connections = 100

    Do I also need to remove the # sign?



  • @adukes40, I only know how with git. Message me and I’ll give you the commands.


  • Senior Developer

    @adukes40 The RC working is done purely through git as branching is greatly simplified. So svn up will not work in your case.

    I dont know the os fog is running on but it normally is found in /etc/my.cnf or /etc/mysql/my.cnf but your location might vary.

    I’d say start by setting max_connections to 300. Once you make the change you will have to restart the MySQL service.



  • Ok, so I can just SVN up? and Where would I increase max_connections, and by how much?


  • Senior Developer

    @adukes40 can you increase your MySQL max_connections? I’m also going to remove persistent connections from the main and addressed the error of calling method debug statically when it is not a static method. I doubt any one of these things will fix what you’re seeing but based on the information I think your MySQL is just not responding to some connections. This makes php wait which would also explain the Max workers issue you’re seeing. You can try running the rc-11 build to see if it helps out of the box. For this I leave someone else to direct as explaining is a bit hard over a 'mobile phone.



  • On a side note, It seems the remote sites cant get past network boot. If I try to boot a machine from the local site where the master server is, it will work, and get to the FOG pxe screen, but its does take longer than normal.



  • @Sebastian-Roth I just learned how to tail a file and dump it to a file last night at class lol. IT WORKED. (sorry small linux golfclap for myself) Anyway here is the last 25 lines of the error.log

    [Tue Sep 13 07:42:16.555758 2016] [:error] [pid 15621] [client 10.103.69.25:50786] PHP Warning: array_combine(): Both parameters should have an equal number of elements in /var/www/html/fog/lib/fog/fogbase.class.php on line 899
    [Tue Sep 13 07:42:16.594268 2016] [:error] [pid 15532] [client 10.103.12.229:62782] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
    [Tue Sep 13 07:42:17.556278 2016] [:error] [pid 15742] [client 10.103.65.30:50729] PHP Warning: PDO::query(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 196
    [Tue Sep 13 07:42:17.556331 2016] [:error] [pid 15742] [client 10.103.65.30:50729] PHP Warning: PDO::query(): Error reading result set’s header in /var/www/html/fog/lib/db/pdodb.class.php on line 196
    [Tue Sep 13 07:42:17.556427 2016] [:error] [pid 15742] [client 10.103.65.30:50729] PHP Strict Standards: Non-static method FOGBase::debug() should not be called statically in /var/www/html/fog/lib/db/pdodb.class.php on line 208
    [Tue Sep 13 07:42:17.897870 2016] [:error] [pid 15472] [client 10.104.12.57:50771] PHP Warning: PDO::query(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 196
    [Tue Sep 13 07:42:17.897931 2016] [:error] [pid 15472] [client 10.104.12.57:50771] PHP Warning: PDO::query(): Error reading result set’s header in /var/www/html/fog/lib/db/pdodb.class.php on line 196
    [Tue Sep 13 07:42:17.898069 2016] [:error] [pid 15472] [client 10.104.12.57:50771] PHP Strict Standards: Non-static method FOGBase::debug() should not be called statically in /var/www/html/fog/lib/db/pdodb.class.php on line 208
    [Tue Sep 13 07:42:17.940687 2016] [:error] [pid 15710] [client 10.103.66.15:50724] PHP Warning: PDO::query(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 196
    [Tue Sep 13 07:42:17.940745 2016] [:error] [pid 15710] [client 10.103.66.15:50724] PHP Warning: PDO::query(): Error reading result set’s header in /var/www/html/fog/lib/db/pdodb.class.php on line 196
    [Tue Sep 13 07:42:17.940827 2016] [:error] [pid 15710] [client 10.103.66.15:50724] PHP Strict Standards: Non-static method FOGBase::debug() should not be called statically in /var/www/html/fog/lib/db/pdodb.class.php on line 208
    [Tue Sep 13 07:42:18.074087 2016] [:error] [pid 15671] [client 10.103.13.134:50659] PHP Warning: PDO::query(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 196
    [Tue Sep 13 07:42:18.074205 2016] [:error] [pid 15671] [client 10.103.13.134:50659] PHP Warning: PDO::query(): Error reading result set’s header in /var/www/html/fog/lib/db/pdodb.class.php on line 196
    [Tue Sep 13 07:42:18.074304 2016] [:error] [pid 15671] [client 10.103.13.134:50659] PHP Strict Standards: Non-static method FOGBase::debug() should not be called statically in /var/www/html/fog/lib/db/pdodb.class.php on line 208
    [Tue Sep 13 07:42:19.016080 2016] [:error] [pid 15558] [client 10.105.65.97:61732] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
    [Tue Sep 13 07:42:19.312576 2016] [:error] [pid 15577] [client 10.103.67.98:50743] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
    [Tue Sep 13 07:42:19.724552 2016] [:error] [pid 15773] [client 10.102.12.235:50724] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
    [Tue Sep 13 07:42:19.824479 2016] [:error] [pid 15667] [client 10.103.14.42:50786] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 152
    [Tue Sep 13 07:42:21.895462 2016] [mpm_prefork:notice] [pid 15466] AH00169: caught SIGTERM, shutting down
    [Tue Sep 13 07:42:41.586125 2016] [mpm_prefork:notice] [pid 1098] AH00163: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f configured – resuming normal operations
    [Tue Sep 13 07:42:41.587691 2016] [core:notice] [pid 1098] AH00094: Command line: ‘/usr/sbin/apache2’
    [Tue Sep 13 07:42:55.604072 2016] [mpm_prefork:error] [pid 1098] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
    [Tue Sep 13 08:47:15.270770 2016] [mpm_prefork:notice] [pid 1105] AH00163: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f configured – resuming normal operations
    [Tue Sep 13 08:47:15.359753 2016] [core:notice] [pid 1105] AH00094: Command line: ‘/usr/sbin/apache2’
    [Tue Sep 13 08:47:31.006640 2016] [mpm_prefork:error] [pid 1105] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting


  • Senior Developer

    @adukes40 I imagine what you’re seeing with the excess lines is specific to the fqdn not being set and that it’s defaulting to 127.0.1.1 as the hostname



  • ran df -h still showing 97G free on main drive. I will look for those logs here in a minute


Log in to reply
 

462
Online

6.1k
Users

13.5k
Topics

127.2k
Posts