#!db


  • Moderator

    Running trunk 8157, just added the 12th storage node in the fog system we are building district wide.

    I’m getting this #!db in the storage pie for one node, the newest one.

    additionally, I can connect from this node via CLI to the main server like this:

    mysql -u fogstorage -pMyPassword -h 10.51.1.53 -D fog
    

    I thought, maybe connections are maxed out. So I ran this to check:

    MariaDB [(none)]> show status like 'Conn%';
    +---------------+-------+
    | Variable_name | Value |
    +---------------+-------+
    | Connections   | 3007  |
    +---------------+-------+
    1 row in set (0.00 sec)
    

    3007 seems like a lot to me, and it keeps going higher by one ever minute or so. Don’t know what’s up with that.

    I set global max connections to 3500 (absurd, yes) just to see if it would clear up, it did not.

    SET GLOBAL max_connections = 3500;
    flush hosts;
    

    I’ve also checked this node’s config.class.php file and it looks to be correct, it’s pointing to the main, using the right DB, username, password.

    This is a head scratcher… Open to suggestions.


  • Moderator

    Tom figured out the problem was due to mis-matching storage node versions and main server version of fog. the main server is behind a little from the recent node installations. Tom was able to create a patch that solved the issue without me having to update the main server, it is working good.


  • Moderator

    I’m unable to replicate this issue at home.


  • Moderator

    @Tom-Elliott if this isn’t figured out by tomorrow morning, I’m going to pull whatever SVN is 8157 and see if that works tomorrow morning, which I am sure will work, and then I’'ll slowly iterate up through the revisions until I find where it breaks.


  • Moderator

    @Tom-Elliott I have firewall configured the same on all the nodes. SELinux is set as Permissive across the board.


  • Senior Developer

    @Wayne-Workman Unlikely, but I really don’t know. That’s why i ask to double verify the connection as it’s being used by the config.class.php and that same node. Maybe SELINUX? Maybe Firewall? I don’t know.


  • Moderator

    @Tom-Elliott For this setup, it’s all CentOS 7.

    We have a little over 400 hosts at the moment (I ended up not doing that import earlier).

    However, this is our 12th and 13th nodes, both installed today, both with this issue. Could it be the number of nodes breaking it?


  • Senior Developer

    @Wayne-Workman I think the exact connection you’re using in the config.class.php and that node connecting remotly to the main db using those connections.

    Did this occur after a specific number of hosts? Meaning @Raymond-Bell have you checked all the same as well? What OS are you using @Raymond-Bell (I know @Wayne-Workman you’re using Fedora or Centos.)?


  • Moderator

    @Tom-Elliott I’ve looked at the config.class.php, it looks all correct.

    What do you mean by connecting? FTP is working fine, it’s replicating.

    Also as a side note, I’m sort of in disbelief that I didn’t include Apache errors nor MySQL errors… I’ll get those tomorrow morning.


  • Senior Developer

    @Wayne-Workman Is the node connecting to the main server? What’s in the config.class.php?


  • Moderator

    @Raymond-Bell Glad to know I’m not the only one.


  • Testers

    I am seeing this also FYI

    0_1466449401116_Untitled3.png


  • Moderator

    also confirmed httpd is running on the node, also gave it a restart too for good measure.


  • Moderator

    All MySQL processes on the server don’t look overwhelming either…

    MariaDB [fog]> show processlist;
    +------+------------+------------------+------+---------+------+-------+------------------+----------+
    | Id   | User       | Host             | db   | Command | Time | State | Info             | Progress |
    +------+------------+------------------+------+---------+------+-------+------------------+----------+
    |   96 | fogstorage | 10.21.1.11:41399 | fog  | Sleep   |    4 |       | NULL             |    0.000 |
    |   97 | fogstorage | 10.21.1.11:41403 | fog  | Sleep   |  217 |       | NULL             |    0.000 |
    |   99 | fogstorage | 10.21.1.11:41407 | fog  | Sleep   |  215 |       | NULL             |    0.000 |
    |  100 | fogstorage | 10.21.1.11:41411 | fog  | Sleep   |   30 |       | NULL             |    0.000 |
    |  101 | fogstorage | 10.21.1.11:41415 | fog  | Sleep   |  209 |       | NULL             |    0.000 |
    | 2520 | fogstorage | 10.11.1.11:52437 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
    | 2521 | fogstorage | 10.11.1.11:52436 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
    | 2522 | fogstorage | 10.11.1.11:52439 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
    | 2523 | fogstorage | 10.11.1.11:52440 | fog  | Sleep   |   30 |       | NULL             |    0.000 |
    | 2524 | fogstorage | 10.11.1.11:52441 | fog  | Sleep   |   20 |       | NULL             |    0.000 |
    | 2525 | fogstorage | 10.11.1.11:52442 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
    | 2542 | fogstorage | 10.21.1.11:41428 | fog  | Sleep   |   20 |       | NULL             |    0.000 |
    | 2543 | fogstorage | 10.21.1.11:41429 | fog  | Sleep   |   20 |       | NULL             |    0.000 |
    | 2544 | fogstorage | 10.21.1.11:41430 | fog  | Sleep   |   10 |       | NULL             |    0.000 |
    | 2545 | fogstorage | 10.26.1.11:35473 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
    | 2546 | fogstorage | 10.26.1.11:35474 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
    | 2547 | fogstorage | 10.26.1.11:35475 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2548 | fogstorage | 10.27.1.11:46449 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2549 | fogstorage | 10.27.1.11:46450 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
    | 2550 | fogstorage | 10.27.1.11:46451 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2554 | fogstorage | 10.29.1.11:35406 | fog  | Sleep   |   27 |       | NULL             |    0.000 |
    | 2555 | fogstorage | 10.29.1.11:35407 | fog  | Sleep   |   28 |       | NULL             |    0.000 |
    | 2556 | fogstorage | 10.29.1.11:35408 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
    | 2560 | fogstorage | 10.21.1.11:41431 | fog  | Sleep   |   20 |       | NULL             |    0.000 |
    | 2561 | fogstorage | 10.21.1.11:41432 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
    | 2562 | fogstorage | 10.26.1.11:35476 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
    | 2563 | fogstorage | 10.26.1.11:35477 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
    | 2564 | fogstorage | 10.27.1.11:46452 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2565 | fogstorage | 10.27.1.11:46453 | fog  | Sleep   |   29 |       | NULL             |    0.000 |
    | 2566 | fogstorage | 10.27.1.11:46454 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
    | 2569 | fogstorage | 10.29.1.11:35409 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
    | 2570 | fogstorage | 10.29.1.11:35410 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
    | 2573 | fogstorage | 10.26.1.11:35478 | fog  | Sleep   |   29 |       | NULL             |    0.000 |
    | 2575 | fogstorage | 10.21.1.11:41433 | fog  | Sleep   |   10 |       | NULL             |    0.000 |
    | 2576 | fogstorage | 10.29.1.11:35411 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
    | 2599 | fogstorage | 10.11.1.11:52443 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
    | 2603 | fogstorage | 10.31.1.11:49498 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
    | 2604 | fogstorage | 10.31.1.11:49499 | fog  | Sleep   |   65 |       | NULL             |    0.000 |
    | 2605 | fogstorage | 10.31.1.11:49500 | fog  | Sleep   |  365 |       | NULL             |    0.000 |
    | 2606 | fogstorage | 10.31.1.11:49501 | fog  | Sleep   |   55 |       | NULL             |    0.000 |
    | 2607 | fogstorage | 10.31.1.11:49502 | fog  | Sleep   |  365 |       | NULL             |    0.000 |
    | 2608 | fogstorage | 10.31.1.11:49503 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2609 | fogstorage | 10.31.1.11:49504 | fog  | Sleep   |   16 |       | NULL             |    0.000 |
    | 2614 | fogstorage | 10.31.1.11:49556 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
    | 2615 | fogstorage | 10.31.1.11:49557 | fog  | Sleep   |   27 |       | NULL             |    0.000 |
    | 2617 | fogstorage | 10.31.1.11:49558 | fog  | Sleep   |    6 |       | NULL             |    0.000 |
    | 2619 | fogstorage | 10.31.1.11:49559 | fog  | Sleep   |    6 |       | NULL             |    0.000 |
    | 2623 | fogstorage | 10.31.1.11:49560 | fog  | Sleep   |    6 |       | NULL             |    0.000 |
    | 2663 | fogstorage | 10.27.1.11:46455 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
    | 2665 | fogstorage | 10.21.1.11:41464 | fog  | Sleep   |   30 |       | NULL             |    0.000 |
    | 2666 | fogstorage | 10.26.1.11:35479 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
    | 2667 | fogstorage | 10.29.1.11:35412 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
    | 2674 | fogstorage | 10.29.1.11:35413 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
    | 2753 | fogstorage | 10.13.1.11:48370 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
    | 2754 | fogstorage | 10.13.1.11:48371 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
    | 2755 | fogstorage | 10.13.1.11:48372 | fog  | Sleep   |   28 |       | NULL             |    0.000 |
    | 2756 | fogstorage | 10.13.1.11:48373 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
    | 2757 | fogstorage | 10.13.1.11:48374 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
    | 2758 | fogstorage | 10.13.1.11:48375 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
    | 2831 | fogstorage | 10.13.1.11:48376 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
    | 2978 | fogstorage | 10.12.1.11:48153 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2979 | fogstorage | 10.12.1.11:48154 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2980 | fogstorage | 10.12.1.11:48155 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2981 | fogstorage | 10.12.1.11:48156 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2982 | fogstorage | 10.12.1.11:48157 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 2983 | fogstorage | 10.12.1.11:48158 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
    | 3053 | fogstorage | 10.12.1.11:48239 | fog  | Query   |    0 | NULL  | show processlist |    0.000 |
    +------+------------+------------------+------+---------+------+-------+------------------+----------+
    67 rows in set (0.00 sec)
    

    Also, number of connections doesn’t look so high either:

    MariaDB [fog]> show status like "%onn%";
    +--------------------------+-------+
    | Variable_name            | Value |
    +--------------------------+-------+
    | Aborted_connects         | 4     |
    | Connections              | 3056  |
    | Max_used_connections     | 106   |
    | Ssl_client_connects      | 0     |
    | Ssl_connect_renegotiates | 0     |
    | Ssl_finished_connects    | 0     |
    | Threads_connected        | 83    |
    +--------------------------+-------+
    7 rows in set (0.00 sec)
    

  • Moderator

    @Tom-Elliott here:
    0_1466445072902_#!db.png


  • Senior Developer

    @Wayne-Workman when where, exactly, are you seeing the #!db?


  • Moderator

    @Tom-Elliott Nothing, connecting manually via CLI works from this node.


  • Senior Developer

    @Wayne-Workman If you try to connect to the db, what does it tell you?


Log in to reply
 

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.