Storage node added - But it's not in the list of nodes
-
@kafluke Do you see any message in the apache error logs when doing this? See my signature on where to find the logs.
-
error_log:
[Sun Apr 07 03:08:03.560444 2019] [auth_digest:notice] [pid 49472] AH01757: generating secret for digest authentication ... [Sun Apr 07 03:08:03.567275 2019] [lbmethod_heartbeat:notice] [pid 49472] AH02282: No slotmem from mod_heartmonitor [Sun Apr 07 03:08:04.240105 2019] [mpm_prefork:notice] [pid 49472] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/7.2.16 configured -- resuming normal operations [Sun Apr 07 03:08:04.240178 2019] [core:notice] [pid 49472] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Mon Apr 08 13:36:17.462416 2019] [proxy_fcgi:error] [pid 61445] [client 192.168.2.137:65018] AH01068: Got bogus version 105, referer: http://192.168.2.75/fog/management/index.php?node=host&sub=list [Wed Apr 10 08:36:07.824757 2019] [proxy_fcgi:error] [pid 55545] [client 192.168.1.14:49812] AH01067: Failed to read entire header got 4 wanted 8 [Wed Apr 10 08:36:07.824834 2019] [proxy_fcgi:error] [pid 55545] (22)Invalid argument: [client 192.168.1.14:49812] AH01075: Error dispatching request to : [Wed Apr 10 15:00:45.512847 2019] [proxy_fcgi:error] [pid 11943] (104)Connection reset by peer: [client 192.168.1.87:61627] AH01075: Error dispatching request to : [Wed Apr 10 15:00:45.622094 2019] [proxy:error] [pid 5975] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed [Wed Apr 10 15:00:45.622198 2019] [proxy_fcgi:error] [pid 5975] [client 192.168.1.87:61628] AH01079: failed to make connection to backend: 127.0.0.1 [Wed Apr 10 15:01:18.714531 2019] [proxy_fcgi:error] [pid 63224] (104)Connection reset by peer: [client 192.168.46.34:58535] AH01075: Error dispatching request to : [Wed Apr 10 15:01:18.715757 2019] [proxy_fcgi:error] [pid 7276] (104)Connection reset by peer: [client 192.168.6.25:54295] AH01075: Error dispatching request to : [Wed Apr 10 15:01:18.716404 2019] [proxy_fcgi:error] [pid 11943] (104)Connection reset by peer: [client 192.168.0.182:55258] AH01075: Error dispatching request to : [Wed Apr 10 15:01:18.722587 2019] [mpm_prefork:notice] [pid 49472] AH00170: caught SIGWINCH, shutting down gracefully [Wed Apr 10 15:01:28.294688 2019] [core:notice] [pid 19667] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0 [Wed Apr 10 15:01:28.301745 2019] [suexec:notice] [pid 19667] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using fe80::aa5d:527d:cf3f:18b6. Set the 'ServerName' directive globally to suppress this message [Wed Apr 10 15:01:28.626397 2019] [auth_digest:notice] [pid 19667] AH01757: generating secret for digest authentication ... [Wed Apr 10 15:01:28.629092 2019] [lbmethod_heartbeat:notice] [pid 19667] AH02282: No slotmem from mod_heartmonitor [Wed Apr 10 15:01:28.711599 2019] [mpm_prefork:notice] [pid 19667] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/7.2.17 configured -- resuming normal operations [Wed Apr 10 15:01:28.711700 2019] [core:notice] [pid 19667] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Wed Apr 10 15:02:42.680530 2019] [proxy:error] [pid 19681] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed [Wed Apr 10 15:02:42.680706 2019] [proxy_fcgi:error] [pid 19681] [client 192.168.2.107:51902] AH01079: failed to make connection to backend: 127.0.0.1 [Wed Apr 10 15:02:42.697823 2019] [proxy:error] [pid 21989] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed [Wed Apr 10 15:02:42.697943 2019] [proxy_fcgi:error] [pid 21989] [client 192.168.2.107:51904] AH01079: failed to make connection to backend: 127.0.0.1 [Wed Apr 10 15:02:43.297983 2019] [proxy:error] [pid 19741] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed [Wed Apr 10 15:02:43.298127 2019] [proxy_fcgi:error] [pid 19741] [client 192.168.12.118:61763] AH01079: failed to make connection to backend: 127.0.0.1 [Wed Apr 10 15:02:43.503995 2019] [proxy:error] [pid 20298] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed [Wed Apr 10 15:02:43.574231 2019] [proxy_fcgi:error] [pid 20298] [client 192.168.12.118:61764] AH01079: failed to make connection to backend: 127.0.0.1 [Wed Apr 10 15:02:43.652284 2019] [proxy:error] [pid 19686] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed [Wed Apr 10 15:02:43.652421 2019] [proxy_fcgi:error] [pid 19686] [client 192.168.12.118:61765] AH01079: failed to make connection to backend: 127.0.0.1 [Wed Apr 10 15:02:43.755951 2019] [proxy:error] [pid 19684] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:9000 (*) failed [Wed Apr 10 15:02:43.764106 2019] [proxy_fcgi:error] [pid 19684] [client 192.168.29.23:49977] AH01079: failed to make connection to backend: 127.0.0.1 [Wed Apr 10 15:02:43.838116 2019] [mpm_prefork:notice] [pid 19667] AH00170: caught SIGWINCH, shutting down gracefully [Wed Apr 10 16:30:20.747680 2019] [core:notice] [pid 5090] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0 [Wed Apr 10 16:30:20.802020 2019] [suexec:notice] [pid 5090] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using fe80::aa5d:527d:cf3f:18b6. Set the 'ServerName' directive globally to suppress this message [Wed Apr 10 16:30:21.130633 2019] [auth_digest:notice] [pid 5090] AH01757: generating secret for digest authentication ... [Wed Apr 10 16:30:21.133799 2019] [lbmethod_heartbeat:notice] [pid 5090] AH02282: No slotmem from mod_heartmonitor [Wed Apr 10 16:30:21.671293 2019] [mpm_prefork:notice] [pid 5090] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/7.2.17 configured -- resuming normal operations [Wed Apr 10 16:30:21.671336 2019] [core:notice] [pid 5090] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'php-fpm log:
https://pastebin.com/ifjwE5SJ -
@kafluke Please run the following commands to see if there is something strange in the database:
shell> mysql -u root -p Password: ... mysql> use fog; ... mysql> SELECT * FROM nfsGroupMembers; ... mysql> SELECT * FROM nfsGroups; ...Post results here.
-
MariaDB [fog]> SELECT * FROM nfsGroupMembers; +-------+------------------------+-------------------------------------+-----------------+------------+-------------+----------------------+------------+---------------+------------------+--------------+--------------+---------------+-------------------+---------+----------------------------------------------+--------+--------------+-----------------+------------+ | ngmID | ngmMemberName | ngmMemberDescription | ngmIsMasterNode | ngmGroupID | ngmRootPath | ngmSSLPath | ngmFTPPath | ngmMaxBitrate | ngmSnapinPath | ngmIsEnabled | ngmHostname | ngmMaxClients | ngmBandwidthLimit | ngmUser | ngmPass | ngmKey | ngmInterface | ngmGraphEnabled | ngmWebroot | +-------+------------------------+-------------------------------------+-----------------+------------+-------------+----------------------+------------+---------------+------------------+--------------+--------------+---------------+-------------------+---------+----------------------------------------------+--------+--------------+-----------------+------------+ | 1 | Park City Fog Server | Auto generated fog nfs group member | 1 | 1 | /images | /opt/fog/snapins/ssl | /images | | /opt/fog/snapins | 1 | 192.168.2.75 | 10 | 0 | fog | 8o0qgd/YwBipfLd5zr8wOKOpJZ09wn4oyljNGMdjZ/Q= | | em1 | 1 | /fog | | 17 | Park City Storage Node | | 0 | 1 | /images | /opt/fog/snapins/ssl | /images | | /opt/fog/snapins | 1 | 192.168.2.76 | 10 | 10000 | fog | 8eYU5hbZj1P63WmOCmU3sPGISCT6oZW8YkhU02LRbw4= | | ens160 | 1 | /fog | | 18 | Buffalo Storage Node | | 0 | 1 | /images | /opt/fog/snapins/ssl | /images | | /opt/fog/snapins | 1 | 192.168.0.75 | 10 | 0 | fog | GJ8ru+eIpsMzyuTPWx38DtdWYPO7Aan7oC3yYAWCjvs= | | ens160 | 1 | /fog | +-------+------------------------+-------------------------------------+-----------------+------------+-------------+----------------------+------------+---------------+------------------+--------------+--------------+---------------+-------------------+---------+----------------------------------------------+--------+--------------+-----------------+------------+ 3 rows in set (0.00 sec) MariaDB [fog]> SELECT * FROM nfsGroups; +------+----------------------+------------------------------+ | ngID | ngName | ngDesc | +------+----------------------+------------------------------+ | 1 | Park City Fog Server | Auto generated fog nfs group | +------+----------------------+------------------------------+ 1 row in set (0.00 sec) MariaDB [fog]> -
@kafluke So do you still see the deleted node in the mysql/mariadb output??
Please help us with some more context. Which node (name) did you delete?
-
@Sebastian-Roth The name of the deleted node was “Dallas Storage Node”. I don’t see it below.
-
@kafluke See if you can add the new storage node definition into the DB manually. I know this is ugly but we ought to see if this is working.
shell> mysql -u root -p Password: ... mysql> use fog; ... mysql> INSERT INTO nfsGroupMembers VALUES ('', 'Dallas Storage Node', '', 0, 1, '/images', '/opt/fog/snapins/ssl', '/images', '', '/opt/fog/snapins', 1, '192.168.X.Y', 10, 0, 'fog', 'password', '', 'ens160', 1, '/fog'); ...Make sure you use the correct IP address. Do not just copy & paste the command you see above. Most values are fine but definitely take a look at IP and maybe name…
-
@Sebastian-Roth Thanks I’ll try that if my VM import doesn’t work. We had a backup of the FOG Node that I moved. It won’t be done until tomorrow. Thanks for the responses.
-
@Sebastian-Roth Manually added it using your commands and now it shows up in the Fog Node list, however, I cannot make any changes to the node. Nothing I select sticks after I click on update. I go back and the change I made is back to what it was before.
Also, I have nothing in my dropdown list for the log viewer. Something got really messed up here.
-
Yeah, something’s definitely jacked. No snapins work any longer. That’s okay, we’ve been meaning to build a new FOG server on one of our ESXi hosts anyway. Now I just hope I can figure out how to migrate all the settings.
-
@kafluke Sounds really strange like if the table is read only. Don’t think I have seen this before.
@george1421 ???
-
@Sebastian-Roth Okay so I just did a complete fresh install on a brand new server. Dev version 1.5.5.3. Absolutely no logs for me to look at inside the log viewer. Weird.