Replication has stopped after ugprade
-
Replication to all my storage nodes has stopped after upgrading.
The log says this (as of 3/6/16):
[02-29-16 1:17:28 pm] * There is nothing to replicate [02-29-16 1:17:28 pm] | We are node name: HQ [02-29-16 1:17:28 pm] * We have node ID: #1 [02-29-16 1:17:28 pm] | We are group name: HQ [02-29-16 1:17:28 pm] * We are group ID: #1 [02-29-16 1:17:28 pm] * Starting Snapin Replication. [02-29-16 1:17:28 pm] * Starting service loop [02-29-16 1:17:28 pm] * Checking for new items every 600 seconds [02-29-16 1:17:28 pm] * Starting SnapinReplicator Service [02-29-16 1:17:28 pm] Interface Ready with IP Address: 192.168.1.66 [02-29-16 1:17:28 pm] Interface Ready with IP Address: xxx.xxx.xxx.xxx ########################################### # GNU GPL Version 3 # # http://fogproject.org/credits # # Credits: # # Free Computer Imaging Solution # ########################################### ```I’ve verified that the replication service is running, but other than that, don’t know what to try.
If it helps, here’s my Apache error.log:
[Sun Mar 06 22:41:26.217550 2016] [:error] [pid 8354] [client 192.168.1.66:46176] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:26.228940 2016] [:error] [pid 8354] [client 192.168.1.66:46176] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:26.294763 2016] [:error] [pid 6838] [client 192.168.1.66:46178] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:26.302681 2016] [:error] [pid 6838] [client 192.168.1.66:46178] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:27.232300 2016] [:error] [pid 6837] [client 192.168.1.66:46182] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:27.241590 2016] [:error] [pid 6837] [client 192.168.1.66:46182] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:27.282051 2016] [:error] [pid 7354] [client 192.168.1.66:46184] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:27.288794 2016] [:error] [pid 7354] [client 192.168.1.66:46184] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:28.206835 2016] [:error] [pid 6836] [client 192.168.1.66:46187] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:28.214122 2016] [:error] [pid 6836] [client 192.168.1.66:46187] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:29.079099 2016] [:error] [pid 6922] [client 192.168.1.66:46190] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:29.088834 2016] [:error] [pid 6922] [client 192.168.1.66:46190] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:30.038310 2016] [:error] [pid 6839] [client 192.168.1.66:46193] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:30.047557 2016] [:error] [pid 6839] [client 192.168.1.66:46193] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:31.003163 2016] [:error] [pid 6867] [client 192.168.1.66:46196] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:31.016484 2016] [:error] [pid 6867] [client 192.168.1.66:46196] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:32.006807 2016] [:error] [pid 6839] [client 192.168.1.66:46199] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:32.014277 2016] [:error] [pid 6839] [client 192.168.1.66:46199] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:33.038982 2016] [:error] [pid 6922] [client 192.168.1.66:46202] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:33.048713 2016] [:error] [pid 6922] [client 192.168.1.66:46202] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:34.025220 2016] [:error] [pid 6867] [client 192.168.1.66:46205] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:34.033522 2016] [:error] [pid 6867] [client 192.168.1.66:46205] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:35.006116 2016] [:error] [pid 7354] [client 192.168.1.66:46208] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:35.013318 2016] [:error] [pid 7354] [client 192.168.1.66:46208] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:35.931800 2016] [:error] [pid 6836] [client 192.168.1.66:46211] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:35.939380 2016] [:error] [pid 6836] [client 192.168.1.66:46211] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:36.923944 2016] [:error] [pid 6837] [client 192.168.1.66:46214] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:36.931825 2016] [:error] [pid 6837] [client 192.168.1.66:46214] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:38.005875 2016] [:error] [pid 6867] [client 192.168.1.66:46217] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:38.016775 2016] [:error] [pid 6867] [client 192.168.1.66:46217] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:39.005655 2016] [:error] [pid 6922] [client 192.168.1.66:46220] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:39.015199 2016] [:error] [pid 6922] [client 192.168.1.66:46220] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:40.055202 2016] [:error] [pid 7354] [client 192.168.1.66:46223] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:40.074702 2016] [:error] [pid 7354] [client 192.168.1.66:46223] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:41.036051 2016] [:error] [pid 8354] [client 192.168.1.66:46226] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:41.045269 2016] [:error] [pid 8354] [client 192.168.1.66:46226] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:41.990316 2016] [:error] [pid 6839] [client 192.168.1.66:46229] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:41.998326 2016] [:error] [pid 6839] [client 192.168.1.66:46229] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:43.010741 2016] [:error] [pid 7354] [client 192.168.1.66:46232] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:43.018896 2016] [:error] [pid 7354] [client 192.168.1.66:46232] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:43.948365 2016] [:error] [pid 6836] [client 192.168.1.66:46235] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:43.965750 2016] [:error] [pid 6836] [client 192.168.1.66:46235] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:44.954691 2016] [:error] [pid 6922] [client 192.168.1.66:46238] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:44.964290 2016] [:error] [pid 6922] [client 192.168.1.66:46238] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:45.993856 2016] [:error] [pid 6840] [client 192.168.1.66:46241] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:46.004338 2016] [:error] [pid 6840] [client 192.168.1.66:46241] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:47.025227 2016] [:error] [pid 7354] [client 192.168.1.66:46244] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:47.036584 2016] [:error] [pid 7354] [client 192.168.1.66:46244] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:47.992503 2016] [:error] [pid 8354] [client 192.168.1.66:46247] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:48.016453 2016] [:error] [pid 8354] [client 192.168.1.66:46247] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:48.965590 2016] [:error] [pid 6838] [client 192.168.1.66:46250] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:48.978061 2016] [:error] [pid 6838] [client 192.168.1.66:46250] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:50.034334 2016] [:error] [pid 6839] [client 192.168.1.66:46253] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:50.044662 2016] [:error] [pid 6839] [client 192.168.1.66:46253] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:50.955111 2016] [:error] [pid 8354] [client 192.168.1.66:46256] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:50.967256 2016] [:error] [pid 8354] [client 192.168.1.66:46256] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:51.982397 2016] [:error] [pid 6867] [client 192.168.1.66:46259] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:51.995166 2016] [:error] [pid 6867] [client 192.168.1.66:46259] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:53.005829 2016] [:error] [pid 6837] [client 192.168.1.66:46262] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:53.023396 2016] [:error] [pid 6837] [client 192.168.1.66:46262] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:53.998062 2016] [:error] [pid 6836] [client 192.168.1.66:46265] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:41:54.021754 2016] [:error] [pid 6836] [client 192.168.1.66:46265] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:41.670099 2016] [:error] [pid 6922] [client 192.168.1.66:46272] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:41.677451 2016] [:error] [pid 6922] [client 192.168.1.66:46272] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:42.665534 2016] [:error] [pid 6836] [client 192.168.1.66:46275] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:42.673665 2016] [:error] [pid 6836] [client 192.168.1.66:46275] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:43.685244 2016] [:error] [pid 6837] [client 192.168.1.66:46278] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:43.692483 2016] [:error] [pid 6837] [client 192.168.1.66:46278] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:44.724118 2016] [:error] [pid 7354] [client 192.168.1.66:46281] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:44.731420 2016] [:error] [pid 7354] [client 192.168.1.66:46281] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:45.839784 2016] [:error] [pid 6836] [client 192.168.1.66:46284] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:45.848039 2016] [:error] [pid 6836] [client 192.168.1.66:46284] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:47.024687 2016] [:error] [pid 8354] [client 192.168.1.66:46287] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:47.031731 2016] [:error] [pid 8354] [client 192.168.1.66:46287] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:49.054243 2016] [:error] [pid 6840] [client 192.168.1.66:46290] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:49.061722 2016] [:error] [pid 6840] [client 192.168.1.66:46290] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:50.075676 2016] [:error] [pid 6837] [client 192.168.1.66:46293] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:50.084335 2016] [:error] [pid 6837] [client 192.168.1.66:46293] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:51.055442 2016] [:error] [pid 6840] [client 192.168.1.66:46296] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:51.067178 2016] [:error] [pid 6840] [client 192.168.1.66:46296] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:52.066709 2016] [:error] [pid 6839] [client 192.168.1.66:46299] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:52.074786 2016] [:error] [pid 6839] [client 192.168.1.66:46299] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:53.115820 2016] [:error] [pid 6922] [client 192.168.1.66:46302] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:53.123291 2016] [:error] [pid 6922] [client 192.168.1.66:46302] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:54.280154 2016] [:error] [pid 6840] [client 192.168.1.66:46305] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:54.287272 2016] [:error] [pid 6840] [client 192.168.1.66:46305] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:55.324572 2016] [:error] [pid 7354] [client 192.168.1.66:46308] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:55.332461 2016] [:error] [pid 7354] [client 192.168.1.66:46308] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:56.364372 2016] [:error] [pid 6867] [client 192.168.1.66:46311] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:42:56.372813 2016] [:error] [pid 6867] [client 192.168.1.66:46311] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:02.869989 2016] [:error] [pid 7354] [client 192.168.1.66:46317] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:02.877448 2016] [:error] [pid 7354] [client 192.168.1.66:46317] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:04.059207 2016] [:error] [pid 11316] [client 192.168.1.66:46320] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:04.066469 2016] [:error] [pid 11316] [client 192.168.1.66:46320] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:05.283028 2016] [:error] [pid 6837] [client 192.168.1.66:46323] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:05.291055 2016] [:error] [pid 6837] [client 192.168.1.66:46323] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:06.366187 2016] [:error] [pid 6867] [client 192.168.1.66:46326] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:06.374776 2016] [:error] [pid 6867] [client 192.168.1.66:46326] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:07.562698 2016] [:error] [pid 6840] [client 192.168.1.66:46329] PHP Warning: fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 [Sun Mar 06 22:43:07.571634 2016] [:error] [pid 6840] [client 192.168.1.66:46329] PHP Warning: fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11 -
After digging a little deeper, it appears that all logs in the log viewer stopped at the 29th of last month, except for apache logs. All of the logs for the storage nodes say “null”.
I’ve verified that both MySQL and FTP passwords for everything are correct.
-
@moses As far as I can tell those messages in the apache error logs are not related to the replication issue. But we should still get that fixed!! I remember talking to Tom because I saw the exact same errors on my virtual machine setup here - this is kind of special as I am using a tap network device and FOG wasn’t happy with that. We fixed it but this might have broken it for others. Can you please post the output of
ip addron your FOG server.@Tom-Elliott Should we better use /sys/class/net/*/operstate instead of calling
ip/ifconfig??About the replication. Tom and Wayne have been working on this lately but I don’t think it’s the same issue. Please check if the services/daemons on your FOG server are all running:
ps ax | grep FOG(maybe they fail on startup/restart?!) -
@Sebastian-Roth This is all I got as output from
ps:1341 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FO TaskScheduler 1364 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts 9595 ? Z 0:00 [FOGPingHosts] <defunct> 9596 ? Z 0:00 [FOGTaskSchedule] <defunct> 10088 pts/1 S+ 0:00 grep --color=auto FOGSo perhaps multiple services are failing on start? Is there any way to find logs that would indicate the why?
-
@Sebastian-Roth Here’s the output of
ip addr:1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 9c:b6:54:f0:3e:0d brd ff:ff:ff:ff:ff:ff inet 192.168.1.66/24 brd 192.168.1.255 scope global em1 valid_lft forever preferred_lft forever inet6 fe80::9eb6:54ff:fef0:3e0d/64 scope link valid_lft forever preferred_lft forever -
@Sebastian-Roth After updating this morning,
the services now appear to be running(they are not running after a restart), but replication still isn’t actually taking place, nor has the log for the Image Replicator changed from my original post. -
@moses can you check the storage nodes all have their proper network interface set correctly? This is why the errors and would also cause issues with the services as they’re waiting for the interface to become available.
-
@Tom-Elliott I checked the setting in Storage Mgmt > Storage Nodes and they all match what is shown using
ifconfigon the nodes thru ssh. -
@moses if you manual stop and start the services do things work?
-
@Tom-Elliott Nope, though the behavior there is kind of odd:
If I restart, it says “failed”.
If I stop, then start, it says “OK” for both, but nothing in the logs, and an image that is supposed to be replicating is not.
-
@moses So you’re running a variant of Ubuntu, Just guessing?
-
@Tom-Elliott ubuntu 14.04
-
@moses Can you run:
sudo service vsftpd stop sudo service FOGImageReplicator stop sudo service FOGSnapinReplicator stop sudo service FOGPingHosts stop sudo service FOGMulticastManager stop sudo service FOGScheduler stop sleep 5 sudo service vsftpd start sudo service FOGImageReplicator start sudo service FOGSnapinReplicator start sudo service FOGPingHosts start sudo service FOGMulticastManager start sudo service FOGScheduler start -
@Tom-Elliott okay, did that:
administrator@SVR-HQ-IMAGING:~$ sudo service vsftpd stop vsftpd stop/waiting administrator@SVR-HQ-IMAGING:~$ sudo service FOGImageReplicator stop * Stopping FOG Computer Imaging Solution: FOGImageReplicator [ OK ] administrator@SVR-HQ-IMAGING:~$ sudo service FOGSnapinReplicator stop * Stopping FOG Computer Imaging Solution: FOGSnapinReplicator [ OK ] administrator@SVR-HQ-IMAGING:~$ sudo service FOGPingHosts stop * Stopping FOG Computer Imaging Solution: FOGPingHosts [ OK ] administrator@SVR-HQ-IMAGING:~$ sudo service FOGMulticastManager stop * Stopping FOG Computer Imaging Solution: FOGMulticastManager [ OK ] administrator@SVR-HQ-IMAGING:~$ sudo service FOGScheduler stop * Stopping FOG Computer Imaging Solution: FOGScheduler [ OK ] administrator@SVR-HQ-IMAGING:~$ sleep 5 administrator@SVR-HQ-IMAGING:~$ sudo service vsftpd start vsftpd start/running, process 9607 administrator@SVR-HQ-IMAGING:~$ sudo service FOGImageReplicator start * Starting FOG Computer Imaging Solution: FOGImageReplicator [ OK ] administrator@SVR-HQ-IMAGING:~$ sudo service FOGSnapinReplicator start * Starting FOG Computer Imaging Solution: FOGSnapinReplicator [ OK ] administrator@SVR-HQ-IMAGING:~$ sudo service FOGPingHosts start * Starting FOG Computer Imaging Solution: FOGPingHosts [ OK ] administrator@SVR-HQ-IMAGING:~$ sudo service FOGMulticastManager start * Starting FOG Computer Imaging Solution: FOGMulticastManager [ OK ] administrator@SVR-HQ-IMAGING:~$ sudo service FOGScheduler startStill no change, however (no replication or log changes)
-
I just updated to the latest, then deleted all images on my slave node, and quickly rebooted both master and slave machines… we’ll see how it goes.
-
@Wayne-Workman See anything on that slave? Trying to determine if this is just an issue somewhere on the master I’m running.
-
@moses It replicated every image back perfectly. There’s something with your setup. Keep in mind it could still be FOG-related though, such as a credentials issue, it could be firewall related, or SELinux related. You might even be out of space on the storage node?? There are many possibilities.
I know you have said you’ve verified the FTP credentials, but let’s make doubly sure? There are instructions here for testing it: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP
Basically you should ssh into your main and try to open an FTP connection to the other nodes using the user/pass they have set in their respective listing under storage management.
Then the opposite, from the nodes, try to open an FTP connection to the main using the main’s credentials that are set in it’s respective Storage Management listing.
-
@moses Did you find out what’s going wrong with the services? Any errors in the apache log when you try starting the service?
We’ve fixed the bandwidth.php issue. Can you please update to the latest version to see if those errors stop and the bandwidth monitor on the dashboard works for you.
-
Unfortunately, even with some hands-on help, I wasn’t able to determine what the cause was. At this point my best bet was that it’s related to Linux. Even upgrading to a newer distro didn’t help. I’m currently in the process of moving my configuration over to CentOS, once I back up my images.
-
@moses Are the services actually running after you started them by hand?
ps ax | grep FOG