#!db

Wayne Workman

Running trunk 8157, just added the 12th storage node in the fog system we are building district wide.

I’m getting this #!db in the storage pie for one node, the newest one.

additionally, I can connect from this node via CLI to the main server like this:

mysql -u fogstorage -pMyPassword -h 10.51.1.53 -D fog

I thought, maybe connections are maxed out. So I ran this to check:

MariaDB [(none)]> show status like 'Conn%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Connections   | 3007  |
+---------------+-------+
1 row in set (0.00 sec)

3007 seems like a lot to me, and it keeps going higher by one ever minute or so. Don’t know what’s up with that.

I set global max connections to 3500 (absurd, yes) just to see if it would clear up, it did not.

SET GLOBAL max_connections = 3500;
flush hosts;

I’ve also checked this node’s config.class.php file and it looks to be correct, it’s pointing to the main, using the right DB, username, password.

This is a head scratcher… Open to suggestions.

Tom Elliott

@Wayne-Workman If you try to connect to the db, what does it tell you?

Wayne Workman

@Tom-Elliott Nothing, connecting manually via CLI works from this node.

Tom Elliott

@Wayne-Workman when where, exactly, are you seeing the #!db?

Wayne Workman

@Tom-Elliott here:
0_1466445072902_#!db.png

Wayne Workman

All MySQL processes on the server don’t look overwhelming either…

MariaDB [fog]> show processlist;
+------+------------+------------------+------+---------+------+-------+------------------+----------+
| Id   | User       | Host             | db   | Command | Time | State | Info             | Progress |
+------+------------+------------------+------+---------+------+-------+------------------+----------+
|   96 | fogstorage | 10.21.1.11:41399 | fog  | Sleep   |    4 |       | NULL             |    0.000 |
|   97 | fogstorage | 10.21.1.11:41403 | fog  | Sleep   |  217 |       | NULL             |    0.000 |
|   99 | fogstorage | 10.21.1.11:41407 | fog  | Sleep   |  215 |       | NULL             |    0.000 |
|  100 | fogstorage | 10.21.1.11:41411 | fog  | Sleep   |   30 |       | NULL             |    0.000 |
|  101 | fogstorage | 10.21.1.11:41415 | fog  | Sleep   |  209 |       | NULL             |    0.000 |
| 2520 | fogstorage | 10.11.1.11:52437 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
| 2521 | fogstorage | 10.11.1.11:52436 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
| 2522 | fogstorage | 10.11.1.11:52439 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
| 2523 | fogstorage | 10.11.1.11:52440 | fog  | Sleep   |   30 |       | NULL             |    0.000 |
| 2524 | fogstorage | 10.11.1.11:52441 | fog  | Sleep   |   20 |       | NULL             |    0.000 |
| 2525 | fogstorage | 10.11.1.11:52442 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
| 2542 | fogstorage | 10.21.1.11:41428 | fog  | Sleep   |   20 |       | NULL             |    0.000 |
| 2543 | fogstorage | 10.21.1.11:41429 | fog  | Sleep   |   20 |       | NULL             |    0.000 |
| 2544 | fogstorage | 10.21.1.11:41430 | fog  | Sleep   |   10 |       | NULL             |    0.000 |
| 2545 | fogstorage | 10.26.1.11:35473 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
| 2546 | fogstorage | 10.26.1.11:35474 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
| 2547 | fogstorage | 10.26.1.11:35475 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2548 | fogstorage | 10.27.1.11:46449 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2549 | fogstorage | 10.27.1.11:46450 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
| 2550 | fogstorage | 10.27.1.11:46451 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2554 | fogstorage | 10.29.1.11:35406 | fog  | Sleep   |   27 |       | NULL             |    0.000 |
| 2555 | fogstorage | 10.29.1.11:35407 | fog  | Sleep   |   28 |       | NULL             |    0.000 |
| 2556 | fogstorage | 10.29.1.11:35408 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
| 2560 | fogstorage | 10.21.1.11:41431 | fog  | Sleep   |   20 |       | NULL             |    0.000 |
| 2561 | fogstorage | 10.21.1.11:41432 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
| 2562 | fogstorage | 10.26.1.11:35476 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
| 2563 | fogstorage | 10.26.1.11:35477 | fog  | Sleep   |   19 |       | NULL             |    0.000 |
| 2564 | fogstorage | 10.27.1.11:46452 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2565 | fogstorage | 10.27.1.11:46453 | fog  | Sleep   |   29 |       | NULL             |    0.000 |
| 2566 | fogstorage | 10.27.1.11:46454 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
| 2569 | fogstorage | 10.29.1.11:35409 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
| 2570 | fogstorage | 10.29.1.11:35410 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
| 2573 | fogstorage | 10.26.1.11:35478 | fog  | Sleep   |   29 |       | NULL             |    0.000 |
| 2575 | fogstorage | 10.21.1.11:41433 | fog  | Sleep   |   10 |       | NULL             |    0.000 |
| 2576 | fogstorage | 10.29.1.11:35411 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
| 2599 | fogstorage | 10.11.1.11:52443 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
| 2603 | fogstorage | 10.31.1.11:49498 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
| 2604 | fogstorage | 10.31.1.11:49499 | fog  | Sleep   |   65 |       | NULL             |    0.000 |
| 2605 | fogstorage | 10.31.1.11:49500 | fog  | Sleep   |  365 |       | NULL             |    0.000 |
| 2606 | fogstorage | 10.31.1.11:49501 | fog  | Sleep   |   55 |       | NULL             |    0.000 |
| 2607 | fogstorage | 10.31.1.11:49502 | fog  | Sleep   |  365 |       | NULL             |    0.000 |
| 2608 | fogstorage | 10.31.1.11:49503 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2609 | fogstorage | 10.31.1.11:49504 | fog  | Sleep   |   16 |       | NULL             |    0.000 |
| 2614 | fogstorage | 10.31.1.11:49556 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
| 2615 | fogstorage | 10.31.1.11:49557 | fog  | Sleep   |   27 |       | NULL             |    0.000 |
| 2617 | fogstorage | 10.31.1.11:49558 | fog  | Sleep   |    6 |       | NULL             |    0.000 |
| 2619 | fogstorage | 10.31.1.11:49559 | fog  | Sleep   |    6 |       | NULL             |    0.000 |
| 2623 | fogstorage | 10.31.1.11:49560 | fog  | Sleep   |    6 |       | NULL             |    0.000 |
| 2663 | fogstorage | 10.27.1.11:46455 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
| 2665 | fogstorage | 10.21.1.11:41464 | fog  | Sleep   |   30 |       | NULL             |    0.000 |
| 2666 | fogstorage | 10.26.1.11:35479 | fog  | Sleep   |    9 |       | NULL             |    0.000 |
| 2667 | fogstorage | 10.29.1.11:35412 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
| 2674 | fogstorage | 10.29.1.11:35413 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
| 2753 | fogstorage | 10.13.1.11:48370 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
| 2754 | fogstorage | 10.13.1.11:48371 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
| 2755 | fogstorage | 10.13.1.11:48372 | fog  | Sleep   |   28 |       | NULL             |    0.000 |
| 2756 | fogstorage | 10.13.1.11:48373 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
| 2757 | fogstorage | 10.13.1.11:48374 | fog  | Sleep   |    7 |       | NULL             |    0.000 |
| 2758 | fogstorage | 10.13.1.11:48375 | fog  | Sleep   |   17 |       | NULL             |    0.000 |
| 2831 | fogstorage | 10.13.1.11:48376 | fog  | Sleep   |   18 |       | NULL             |    0.000 |
| 2978 | fogstorage | 10.12.1.11:48153 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2979 | fogstorage | 10.12.1.11:48154 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2980 | fogstorage | 10.12.1.11:48155 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2981 | fogstorage | 10.12.1.11:48156 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2982 | fogstorage | 10.12.1.11:48157 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 2983 | fogstorage | 10.12.1.11:48158 | fog  | Sleep   |    8 |       | NULL             |    0.000 |
| 3053 | fogstorage | 10.12.1.11:48239 | fog  | Query   |    0 | NULL  | show processlist |    0.000 |
+------+------------+------------------+------+---------+------+-------+------------------+----------+
67 rows in set (0.00 sec)

Also, number of connections doesn’t look so high either:

MariaDB [fog]> show status like "%onn%";
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| Aborted_connects         | 4     |
| Connections              | 3056  |
| Max_used_connections     | 106   |
| Ssl_client_connects      | 0     |
| Ssl_connect_renegotiates | 0     |
| Ssl_finished_connects    | 0     |
| Threads_connected        | 83    |
+--------------------------+-------+
7 rows in set (0.00 sec)

Wayne Workman

also confirmed httpd is running on the node, also gave it a restart too for good measure.

Raymond Bell

I am seeing this also FYI

Wayne Workman

@Raymond-Bell Glad to know I’m not the only one.

Tom Elliott

@Wayne-Workman Is the node connecting to the main server? What’s in the config.class.php?

Wayne Workman

@Tom-Elliott I’ve looked at the config.class.php, it looks all correct.

What do you mean by connecting? FTP is working fine, it’s replicating.

Also as a side note, I’m sort of in disbelief that I didn’t include Apache errors nor MySQL errors… I’ll get those tomorrow morning.

Tom Elliott

@Wayne-Workman I think the exact connection you’re using in the config.class.php and that node connecting remotly to the main db using those connections.

Did this occur after a specific number of hosts? Meaning @Raymond-Bell have you checked all the same as well? What OS are you using @Raymond-Bell (I know @Wayne-Workman you’re using Fedora or Centos.)?

Wayne Workman

@Tom-Elliott For this setup, it’s all CentOS 7.

We have a little over 400 hosts at the moment (I ended up not doing that import earlier).

However, this is our 12th and 13th nodes, both installed today, both with this issue. Could it be the number of nodes breaking it?

Tom Elliott

@Wayne-Workman Unlikely, but I really don’t know. That’s why i ask to double verify the connection as it’s being used by the config.class.php and that same node. Maybe SELINUX? Maybe Firewall? I don’t know.

Wayne Workman

@Tom-Elliott I have firewall configured the same on all the nodes. SELinux is set as Permissive across the board.

Wayne Workman

@Tom-Elliott if this isn’t figured out by tomorrow morning, I’m going to pull whatever SVN is 8157 and see if that works tomorrow morning, which I am sure will work, and then I’'ll slowly iterate up through the revisions until I find where it breaks.

Wayne Workman

I’m unable to replicate this issue at home.

Wayne Workman

Tom figured out the problem was due to mis-matching storage node versions and main server version of fog. the main server is behind a little from the recent node installations. Tom was able to create a patch that solved the issue without me having to update the main server, it is working good.

#!db

250

11.7k

17.1k

154.7k