Replication has stopped after ugprade



  • Replication to all my storage nodes has stopped after upgrading.

    The log says this (as of 3/6/16):

    [02-29-16 1:17:28 pm] * There is nothing to replicate
    [02-29-16 1:17:28 pm] | We are node name: HQ
    [02-29-16 1:17:28 pm] * We have node ID: #1
    [02-29-16 1:17:28 pm] | We are group name: HQ
    [02-29-16 1:17:28 pm] * We are group ID: #1
    [02-29-16 1:17:28 pm] * Starting Snapin Replication.
    [02-29-16 1:17:28 pm] * Starting service loop
    [02-29-16 1:17:28 pm] * Checking for new items every 600 seconds
    [02-29-16 1:17:28 pm] * Starting SnapinReplicator Service
    [02-29-16 1:17:28 pm] Interface Ready with IP Address: 192.168.1.66
    [02-29-16 1:17:28 pm] Interface Ready with IP Address: xxx.xxx.xxx.xxx
    
     ###########################################
     # GNU GPL Version 3 #
     # http://fogproject.org/credits #
     # Credits: #
     # Free Computer Imaging Solution #
     ########################################### ```
    

    I’ve verified that the replication service is running, but other than that, don’t know what to try.

    If it helps, here’s my Apache error.log:

    [Sun Mar 06 22:41:26.217550 2016] [:error] [pid 8354] [client 192.168.1.66:46176] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:26.228940 2016] [:error] [pid 8354] [client 192.168.1.66:46176] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:26.294763 2016] [:error] [pid 6838] [client 192.168.1.66:46178] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:26.302681 2016] [:error] [pid 6838] [client 192.168.1.66:46178] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:27.232300 2016] [:error] [pid 6837] [client 192.168.1.66:46182] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:27.241590 2016] [:error] [pid 6837] [client 192.168.1.66:46182] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:27.282051 2016] [:error] [pid 7354] [client 192.168.1.66:46184] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:27.288794 2016] [:error] [pid 7354] [client 192.168.1.66:46184] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:28.206835 2016] [:error] [pid 6836] [client 192.168.1.66:46187] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:28.214122 2016] [:error] [pid 6836] [client 192.168.1.66:46187] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:29.079099 2016] [:error] [pid 6922] [client 192.168.1.66:46190] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:29.088834 2016] [:error] [pid 6922] [client 192.168.1.66:46190] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:30.038310 2016] [:error] [pid 6839] [client 192.168.1.66:46193] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:30.047557 2016] [:error] [pid 6839] [client 192.168.1.66:46193] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:31.003163 2016] [:error] [pid 6867] [client 192.168.1.66:46196] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:31.016484 2016] [:error] [pid 6867] [client 192.168.1.66:46196] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:32.006807 2016] [:error] [pid 6839] [client 192.168.1.66:46199] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:32.014277 2016] [:error] [pid 6839] [client 192.168.1.66:46199] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:33.038982 2016] [:error] [pid 6922] [client 192.168.1.66:46202] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:33.048713 2016] [:error] [pid 6922] [client 192.168.1.66:46202] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:34.025220 2016] [:error] [pid 6867] [client 192.168.1.66:46205] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:34.033522 2016] [:error] [pid 6867] [client 192.168.1.66:46205] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:35.006116 2016] [:error] [pid 7354] [client 192.168.1.66:46208] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:35.013318 2016] [:error] [pid 7354] [client 192.168.1.66:46208] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:35.931800 2016] [:error] [pid 6836] [client 192.168.1.66:46211] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:35.939380 2016] [:error] [pid 6836] [client 192.168.1.66:46211] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:36.923944 2016] [:error] [pid 6837] [client 192.168.1.66:46214] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:36.931825 2016] [:error] [pid 6837] [client 192.168.1.66:46214] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:38.005875 2016] [:error] [pid 6867] [client 192.168.1.66:46217] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:38.016775 2016] [:error] [pid 6867] [client 192.168.1.66:46217] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:39.005655 2016] [:error] [pid 6922] [client 192.168.1.66:46220] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:39.015199 2016] [:error] [pid 6922] [client 192.168.1.66:46220] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:40.055202 2016] [:error] [pid 7354] [client 192.168.1.66:46223] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:40.074702 2016] [:error] [pid 7354] [client 192.168.1.66:46223] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:41.036051 2016] [:error] [pid 8354] [client 192.168.1.66:46226] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:41.045269 2016] [:error] [pid 8354] [client 192.168.1.66:46226] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:41.990316 2016] [:error] [pid 6839] [client 192.168.1.66:46229] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:41.998326 2016] [:error] [pid 6839] [client 192.168.1.66:46229] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:43.010741 2016] [:error] [pid 7354] [client 192.168.1.66:46232] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:43.018896 2016] [:error] [pid 7354] [client 192.168.1.66:46232] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:43.948365 2016] [:error] [pid 6836] [client 192.168.1.66:46235] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:43.965750 2016] [:error] [pid 6836] [client 192.168.1.66:46235] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:44.954691 2016] [:error] [pid 6922] [client 192.168.1.66:46238] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:44.964290 2016] [:error] [pid 6922] [client 192.168.1.66:46238] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:45.993856 2016] [:error] [pid 6840] [client 192.168.1.66:46241] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:46.004338 2016] [:error] [pid 6840] [client 192.168.1.66:46241] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:47.025227 2016] [:error] [pid 7354] [client 192.168.1.66:46244] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:47.036584 2016] [:error] [pid 7354] [client 192.168.1.66:46244] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:47.992503 2016] [:error] [pid 8354] [client 192.168.1.66:46247] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:48.016453 2016] [:error] [pid 8354] [client 192.168.1.66:46247] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:48.965590 2016] [:error] [pid 6838] [client 192.168.1.66:46250] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:48.978061 2016] [:error] [pid 6838] [client 192.168.1.66:46250] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:50.034334 2016] [:error] [pid 6839] [client 192.168.1.66:46253] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:50.044662 2016] [:error] [pid 6839] [client 192.168.1.66:46253] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:50.955111 2016] [:error] [pid 8354] [client 192.168.1.66:46256] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:50.967256 2016] [:error] [pid 8354] [client 192.168.1.66:46256] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:51.982397 2016] [:error] [pid 6867] [client 192.168.1.66:46259] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:51.995166 2016] [:error] [pid 6867] [client 192.168.1.66:46259] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:53.005829 2016] [:error] [pid 6837] [client 192.168.1.66:46262] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:53.023396 2016] [:error] [pid 6837] [client 192.168.1.66:46262] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:53.998062 2016] [:error] [pid 6836] [client 192.168.1.66:46265] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:41:54.021754 2016] [:error] [pid 6836] [client 192.168.1.66:46265] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:41.670099 2016] [:error] [pid 6922] [client 192.168.1.66:46272] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:41.677451 2016] [:error] [pid 6922] [client 192.168.1.66:46272] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:42.665534 2016] [:error] [pid 6836] [client 192.168.1.66:46275] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:42.673665 2016] [:error] [pid 6836] [client 192.168.1.66:46275] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:43.685244 2016] [:error] [pid 6837] [client 192.168.1.66:46278] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:43.692483 2016] [:error] [pid 6837] [client 192.168.1.66:46278] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:44.724118 2016] [:error] [pid 7354] [client 192.168.1.66:46281] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:44.731420 2016] [:error] [pid 7354] [client 192.168.1.66:46281] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:45.839784 2016] [:error] [pid 6836] [client 192.168.1.66:46284] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:45.848039 2016] [:error] [pid 6836] [client 192.168.1.66:46284] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:47.024687 2016] [:error] [pid 8354] [client 192.168.1.66:46287] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:47.031731 2016] [:error] [pid 8354] [client 192.168.1.66:46287] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:49.054243 2016] [:error] [pid 6840] [client 192.168.1.66:46290] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:49.061722 2016] [:error] [pid 6840] [client 192.168.1.66:46290] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:50.075676 2016] [:error] [pid 6837] [client 192.168.1.66:46293] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:50.084335 2016] [:error] [pid 6837] [client 192.168.1.66:46293] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:51.055442 2016] [:error] [pid 6840] [client 192.168.1.66:46296] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:51.067178 2016] [:error] [pid 6840] [client 192.168.1.66:46296] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:52.066709 2016] [:error] [pid 6839] [client 192.168.1.66:46299] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:52.074786 2016] [:error] [pid 6839] [client 192.168.1.66:46299] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:53.115820 2016] [:error] [pid 6922] [client 192.168.1.66:46302] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:53.123291 2016] [:error] [pid 6922] [client 192.168.1.66:46302] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:54.280154 2016] [:error] [pid 6840] [client 192.168.1.66:46305] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:54.287272 2016] [:error] [pid 6840] [client 192.168.1.66:46305] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:55.324572 2016] [:error] [pid 7354] [client 192.168.1.66:46308] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:55.332461 2016] [:error] [pid 7354] [client 192.168.1.66:46308] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:56.364372 2016] [:error] [pid 6867] [client 192.168.1.66:46311] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:42:56.372813 2016] [:error] [pid 6867] [client 192.168.1.66:46311] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:02.869989 2016] [:error] [pid 7354] [client 192.168.1.66:46317] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:02.877448 2016] [:error] [pid 7354] [client 192.168.1.66:46317] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:04.059207 2016] [:error] [pid 11316] [client 192.168.1.66:46320] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:04.066469 2016] [:error] [pid 11316] [client 192.168.1.66:46320] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:05.283028 2016] [:error] [pid 6837] [client 192.168.1.66:46323] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:05.291055 2016] [:error] [pid 6837] [client 192.168.1.66:46323] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:06.366187 2016] [:error] [pid 6867] [client 192.168.1.66:46326] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:06.374776 2016] [:error] [pid 6867] [client 192.168.1.66:46326] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:07.562698 2016] [:error] [pid 6840] [client 192.168.1.66:46329] PHP Warning:  fopen(/sys/class/net//statistics/rx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    [Sun Mar 06 22:43:07.571634 2016] [:error] [pid 6840] [client 192.168.1.66:46329] PHP Warning:  fopen(/sys/class/net//statistics/tx_bytes): failed to open stream: No such file or directory in /var/www/html/fog/status/bandwidth.php on line 11
    

  • Senior Developer

    @moses in the installer’s bin/error_logs/fog_error_<versionNumberOfInstall>.log what’s at the end of the file?



  • After updating the node in question (despite it failing at “Restarting apache2 for fog vhost”), replication has been fixed.



  • @Sebastian-Roth fopen errors are gone, now I got a shiny new error (repeated many times):

    [Tue Mar 08 13:46:04.596088 2016] [:error] [pid 2547] [client 192.168.1.144:60293] PHP Warning:  mysqli::query(): invalid object or resource mysqli\n in /var/www/html/fog/lib/db/mysql.class.php on line 52, referer: http://192.168.1.66/fog/management/index.php?node=home
    


  • @Sebastian-Roth In this case, it’s preventing the replication service from proceeding any further. Replication never starts, to any nodes. If I remove that particular storage group from that image, replication works find to all the others.

    Checking on that now…


  • Developer

    @moses Thanks for reporting! I am sure Tom will fix this in a second. Anyhow, I don’t think this is a show stopper for you. Usually those “Undefined index” messages are just notices, not real PHP errors. Log should be going on… I hope!

    By the way. Do you still have the fopen warnings in your apache error log? A fix has been pushed this morning.



  • After moving my installation over to CentOS, the Image Replication service now starts. Hooray!

    But alas, I run into another issue. See the last line of the fogreplicator.log:

      ###########################################
      #     Free Computer Imaging Solution      #
      #     Credits:                            #
      #     http://fogproject.org/credits       #
      #     GNU GPL Version 3                   #
      ###########################################
    
    [03-08-16 2:00:59 pm] Interface Ready with IP Address: 8.67.5.309
    [03-08-16 2:00:59 pm] Interface Ready with IP Address: 192.168.1.66
    [03-08-16 2:00:59 pm] Interface Ready with IP Address: SVR-HQ-IMAGING
    [03-08-16 2:00:59 pm]  * Starting ImageReplicator Service
    [03-08-16 2:00:59 pm]  * Checking for new items every 600 seconds
    [03-08-16 2:00:59 pm]  * Starting service loop
    [03-08-16 2:00:59 pm]  * Starting Image Replication.
    [03-08-16 2:00:59 pm]  * We are group ID: #1
    [03-08-16 2:00:59 pm]  | We are group name: HQ
    [03-08-16 2:00:59 pm]  * We have node ID: #1
    [03-08-16 2:00:59 pm]  | We are node name: SVR-HQ-IMAGING
    [03-08-16 2:00:59 pm]  * Found Image to transfer to 3 group(s)
    [03-08-16 2:00:59 pm]  | Image name: Universal-W7P-x64-ENGINEERING
    [03-08-16 2:01:00 pm]  * Type: 8, File: /var/www/html/fog/lib/fog/fogbase.class.php, Line: 55, Message: Undefined index: REQUEST_METHOD
    

    This error only appears if I add an image to one specific storage group. Should I maybe delete the storage group or the node there and re-create them?


  • Developer

    if you run top, how much of your cpu do these services utilize?


  • Developer

    @moses Are the services actually running after you started them by hand? ps ax | grep FOG



  • Unfortunately, even with some hands-on help, I wasn’t able to determine what the cause was. At this point my best bet was that it’s related to Linux. Even upgrading to a newer distro didn’t help. I’m currently in the process of moving my configuration over to CentOS, once I back up my images.


  • Developer

    @moses Did you find out what’s going wrong with the services? Any errors in the apache log when you try starting the service?

    We’ve fixed the bandwidth.php issue. Can you please update to the latest version to see if those errors stop and the bandwidth monitor on the dashboard works for you.


  • Moderator

    @moses It replicated every image back perfectly. There’s something with your setup. Keep in mind it could still be FOG-related though, such as a credentials issue, it could be firewall related, or SELinux related. You might even be out of space on the storage node?? There are many possibilities.

    I know you have said you’ve verified the FTP credentials, but let’s make doubly sure? There are instructions here for testing it: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP

    Basically you should ssh into your main and try to open an FTP connection to the other nodes using the user/pass they have set in their respective listing under storage management.

    Then the opposite, from the nodes, try to open an FTP connection to the main using the main’s credentials that are set in it’s respective Storage Management listing.



  • @Wayne-Workman See anything on that slave? Trying to determine if this is just an issue somewhere on the master I’m running.


  • Moderator

    I just updated to the latest, then deleted all images on my slave node, and quickly rebooted both master and slave machines… we’ll see how it goes. :-)



  • @Tom-Elliott okay, did that:

    administrator@SVR-HQ-IMAGING:~$ sudo service vsftpd stop
    vsftpd stop/waiting
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGImageReplicator stop
     * Stopping FOG Computer Imaging Solution: FOGImageReplicator            [ OK ]
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGSnapinReplicator stop
     * Stopping FOG Computer Imaging Solution: FOGSnapinReplicator           [ OK ]
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGPingHosts stop
     * Stopping FOG Computer Imaging Solution: FOGPingHosts                  [ OK ]
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGMulticastManager stop
     * Stopping FOG Computer Imaging Solution: FOGMulticastManager           [ OK ]
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGScheduler stop
     * Stopping FOG Computer Imaging Solution: FOGScheduler                  [ OK ]
    administrator@SVR-HQ-IMAGING:~$ sleep 5
    administrator@SVR-HQ-IMAGING:~$ sudo service vsftpd start
    vsftpd start/running, process 9607
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGImageReplicator start
     * Starting FOG Computer Imaging Solution: FOGImageReplicator            [ OK ]
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGSnapinReplicator start
     * Starting FOG Computer Imaging Solution: FOGSnapinReplicator           [ OK ]
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGPingHosts start
     * Starting FOG Computer Imaging Solution: FOGPingHosts                  [ OK ]
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGMulticastManager start
     * Starting FOG Computer Imaging Solution: FOGMulticastManager           [ OK ]
    administrator@SVR-HQ-IMAGING:~$ sudo service FOGScheduler start
    

    Still no change, however (no replication or log changes)


  • Senior Developer

    @moses Can you run:

    sudo service vsftpd stop
    sudo service FOGImageReplicator stop
    sudo service FOGSnapinReplicator stop
    sudo service FOGPingHosts stop
    sudo service FOGMulticastManager stop
    sudo service FOGScheduler stop
    sleep 5
    sudo service vsftpd start
    sudo service FOGImageReplicator start
    sudo service FOGSnapinReplicator start
    sudo service FOGPingHosts start
    sudo service FOGMulticastManager start
    sudo service FOGScheduler start
    


  • @Tom-Elliott ubuntu 14.04


  • Senior Developer

    @moses So you’re running a variant of Ubuntu, Just guessing?



  • @Tom-Elliott Nope, though the behavior there is kind of odd:

    If I restart, it says “failed”.

    If I stop, then start, it says “OK” for both, but nothing in the logs, and an image that is supposed to be replicating is not.


  • Senior Developer

    @moses if you manual stop and start the services do things work?


Log in to reply
 

408
Online

39.5k
Users

11.1k
Topics

105.6k
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.