#!db
-
@Wayne-Workman If you try to connect to the db, what does it tell you?
-
@Tom-Elliott Nothing, connecting manually via CLI works from this node.
-
@Wayne-Workman when where, exactly, are you seeing the #!db?
-
@Tom-Elliott here:
-
All MySQL processes on the server don’t look overwhelming either…
MariaDB [fog]> show processlist; +------+------------+------------------+------+---------+------+-------+------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +------+------------+------------------+------+---------+------+-------+------------------+----------+ | 96 | fogstorage | 10.21.1.11:41399 | fog | Sleep | 4 | | NULL | 0.000 | | 97 | fogstorage | 10.21.1.11:41403 | fog | Sleep | 217 | | NULL | 0.000 | | 99 | fogstorage | 10.21.1.11:41407 | fog | Sleep | 215 | | NULL | 0.000 | | 100 | fogstorage | 10.21.1.11:41411 | fog | Sleep | 30 | | NULL | 0.000 | | 101 | fogstorage | 10.21.1.11:41415 | fog | Sleep | 209 | | NULL | 0.000 | | 2520 | fogstorage | 10.11.1.11:52437 | fog | Sleep | 9 | | NULL | 0.000 | | 2521 | fogstorage | 10.11.1.11:52436 | fog | Sleep | 19 | | NULL | 0.000 | | 2522 | fogstorage | 10.11.1.11:52439 | fog | Sleep | 9 | | NULL | 0.000 | | 2523 | fogstorage | 10.11.1.11:52440 | fog | Sleep | 30 | | NULL | 0.000 | | 2524 | fogstorage | 10.11.1.11:52441 | fog | Sleep | 20 | | NULL | 0.000 | | 2525 | fogstorage | 10.11.1.11:52442 | fog | Sleep | 19 | | NULL | 0.000 | | 2542 | fogstorage | 10.21.1.11:41428 | fog | Sleep | 20 | | NULL | 0.000 | | 2543 | fogstorage | 10.21.1.11:41429 | fog | Sleep | 20 | | NULL | 0.000 | | 2544 | fogstorage | 10.21.1.11:41430 | fog | Sleep | 10 | | NULL | 0.000 | | 2545 | fogstorage | 10.26.1.11:35473 | fog | Sleep | 19 | | NULL | 0.000 | | 2546 | fogstorage | 10.26.1.11:35474 | fog | Sleep | 9 | | NULL | 0.000 | | 2547 | fogstorage | 10.26.1.11:35475 | fog | Sleep | 8 | | NULL | 0.000 | | 2548 | fogstorage | 10.27.1.11:46449 | fog | Sleep | 8 | | NULL | 0.000 | | 2549 | fogstorage | 10.27.1.11:46450 | fog | Sleep | 18 | | NULL | 0.000 | | 2550 | fogstorage | 10.27.1.11:46451 | fog | Sleep | 8 | | NULL | 0.000 | | 2554 | fogstorage | 10.29.1.11:35406 | fog | Sleep | 27 | | NULL | 0.000 | | 2555 | fogstorage | 10.29.1.11:35407 | fog | Sleep | 28 | | NULL | 0.000 | | 2556 | fogstorage | 10.29.1.11:35408 | fog | Sleep | 7 | | NULL | 0.000 | | 2560 | fogstorage | 10.21.1.11:41431 | fog | Sleep | 20 | | NULL | 0.000 | | 2561 | fogstorage | 10.21.1.11:41432 | fog | Sleep | 9 | | NULL | 0.000 | | 2562 | fogstorage | 10.26.1.11:35476 | fog | Sleep | 19 | | NULL | 0.000 | | 2563 | fogstorage | 10.26.1.11:35477 | fog | Sleep | 19 | | NULL | 0.000 | | 2564 | fogstorage | 10.27.1.11:46452 | fog | Sleep | 8 | | NULL | 0.000 | | 2565 | fogstorage | 10.27.1.11:46453 | fog | Sleep | 29 | | NULL | 0.000 | | 2566 | fogstorage | 10.27.1.11:46454 | fog | Sleep | 18 | | NULL | 0.000 | | 2569 | fogstorage | 10.29.1.11:35409 | fog | Sleep | 17 | | NULL | 0.000 | | 2570 | fogstorage | 10.29.1.11:35410 | fog | Sleep | 7 | | NULL | 0.000 | | 2573 | fogstorage | 10.26.1.11:35478 | fog | Sleep | 29 | | NULL | 0.000 | | 2575 | fogstorage | 10.21.1.11:41433 | fog | Sleep | 10 | | NULL | 0.000 | | 2576 | fogstorage | 10.29.1.11:35411 | fog | Sleep | 7 | | NULL | 0.000 | | 2599 | fogstorage | 10.11.1.11:52443 | fog | Sleep | 9 | | NULL | 0.000 | | 2603 | fogstorage | 10.31.1.11:49498 | fog | Sleep | 17 | | NULL | 0.000 | | 2604 | fogstorage | 10.31.1.11:49499 | fog | Sleep | 65 | | NULL | 0.000 | | 2605 | fogstorage | 10.31.1.11:49500 | fog | Sleep | 365 | | NULL | 0.000 | | 2606 | fogstorage | 10.31.1.11:49501 | fog | Sleep | 55 | | NULL | 0.000 | | 2607 | fogstorage | 10.31.1.11:49502 | fog | Sleep | 365 | | NULL | 0.000 | | 2608 | fogstorage | 10.31.1.11:49503 | fog | Sleep | 8 | | NULL | 0.000 | | 2609 | fogstorage | 10.31.1.11:49504 | fog | Sleep | 16 | | NULL | 0.000 | | 2614 | fogstorage | 10.31.1.11:49556 | fog | Sleep | 17 | | NULL | 0.000 | | 2615 | fogstorage | 10.31.1.11:49557 | fog | Sleep | 27 | | NULL | 0.000 | | 2617 | fogstorage | 10.31.1.11:49558 | fog | Sleep | 6 | | NULL | 0.000 | | 2619 | fogstorage | 10.31.1.11:49559 | fog | Sleep | 6 | | NULL | 0.000 | | 2623 | fogstorage | 10.31.1.11:49560 | fog | Sleep | 6 | | NULL | 0.000 | | 2663 | fogstorage | 10.27.1.11:46455 | fog | Sleep | 18 | | NULL | 0.000 | | 2665 | fogstorage | 10.21.1.11:41464 | fog | Sleep | 30 | | NULL | 0.000 | | 2666 | fogstorage | 10.26.1.11:35479 | fog | Sleep | 9 | | NULL | 0.000 | | 2667 | fogstorage | 10.29.1.11:35412 | fog | Sleep | 17 | | NULL | 0.000 | | 2674 | fogstorage | 10.29.1.11:35413 | fog | Sleep | 17 | | NULL | 0.000 | | 2753 | fogstorage | 10.13.1.11:48370 | fog | Sleep | 7 | | NULL | 0.000 | | 2754 | fogstorage | 10.13.1.11:48371 | fog | Sleep | 7 | | NULL | 0.000 | | 2755 | fogstorage | 10.13.1.11:48372 | fog | Sleep | 28 | | NULL | 0.000 | | 2756 | fogstorage | 10.13.1.11:48373 | fog | Sleep | 18 | | NULL | 0.000 | | 2757 | fogstorage | 10.13.1.11:48374 | fog | Sleep | 7 | | NULL | 0.000 | | 2758 | fogstorage | 10.13.1.11:48375 | fog | Sleep | 17 | | NULL | 0.000 | | 2831 | fogstorage | 10.13.1.11:48376 | fog | Sleep | 18 | | NULL | 0.000 | | 2978 | fogstorage | 10.12.1.11:48153 | fog | Sleep | 8 | | NULL | 0.000 | | 2979 | fogstorage | 10.12.1.11:48154 | fog | Sleep | 8 | | NULL | 0.000 | | 2980 | fogstorage | 10.12.1.11:48155 | fog | Sleep | 8 | | NULL | 0.000 | | 2981 | fogstorage | 10.12.1.11:48156 | fog | Sleep | 8 | | NULL | 0.000 | | 2982 | fogstorage | 10.12.1.11:48157 | fog | Sleep | 8 | | NULL | 0.000 | | 2983 | fogstorage | 10.12.1.11:48158 | fog | Sleep | 8 | | NULL | 0.000 | | 3053 | fogstorage | 10.12.1.11:48239 | fog | Query | 0 | NULL | show processlist | 0.000 | +------+------------+------------------+------+---------+------+-------+------------------+----------+ 67 rows in set (0.00 sec)
Also, number of connections doesn’t look so high either:
MariaDB [fog]> show status like "%onn%"; +--------------------------+-------+ | Variable_name | Value | +--------------------------+-------+ | Aborted_connects | 4 | | Connections | 3056 | | Max_used_connections | 106 | | Ssl_client_connects | 0 | | Ssl_connect_renegotiates | 0 | | Ssl_finished_connects | 0 | | Threads_connected | 83 | +--------------------------+-------+ 7 rows in set (0.00 sec)
-
also confirmed httpd is running on the node, also gave it a restart too for good measure.
-
I am seeing this also FYI
-
@Raymond-Bell Glad to know I’m not the only one.
-
@Wayne-Workman Is the node connecting to the main server? What’s in the config.class.php?
-
@Tom-Elliott I’ve looked at the config.class.php, it looks all correct.
What do you mean by connecting? FTP is working fine, it’s replicating.
Also as a side note, I’m sort of in disbelief that I didn’t include Apache errors nor MySQL errors… I’ll get those tomorrow morning.
-
@Wayne-Workman I think the exact connection you’re using in the config.class.php and that node connecting remotly to the main db using those connections.
Did this occur after a specific number of hosts? Meaning @Raymond-Bell have you checked all the same as well? What OS are you using @Raymond-Bell (I know @Wayne-Workman you’re using Fedora or Centos.)?
-
@Tom-Elliott For this setup, it’s all CentOS 7.
We have a little over 400 hosts at the moment (I ended up not doing that import earlier).
However, this is our 12th and 13th nodes, both installed today, both with this issue. Could it be the number of nodes breaking it?
-
@Wayne-Workman Unlikely, but I really don’t know. That’s why i ask to double verify the connection as it’s being used by the config.class.php and that same node. Maybe SELINUX? Maybe Firewall? I don’t know.
-
@Tom-Elliott I have firewall configured the same on all the nodes. SELinux is set as Permissive across the board.
-
@Tom-Elliott if this isn’t figured out by tomorrow morning, I’m going to pull whatever SVN is 8157 and see if that works tomorrow morning, which I am sure will work, and then I’'ll slowly iterate up through the revisions until I find where it breaks.
-
I’m unable to replicate this issue at home.
-
Tom figured out the problem was due to mis-matching storage node versions and main server version of fog. the main server is behind a little from the recent node installations. Tom was able to create a patch that solved the issue without me having to update the main server, it is working good.