Fog Replication Not working

[Sebastian Roth]

@Obsidian Great to hear your got it all back together with the help of Wayne!

Please give us a quick update if it all works on Friday so we can mark this solved.

[Obsidian]

Okay so from what I can tell most (all but maybe one site) didn’t replicate all of the or most of the Windows 7 images the one that didn’t replicate I think is a pw issue so am not overtly concerned on it at this time. However I recaptured my Windows 10 image last week and so far at this time it has still not replicated. I looked at the sync logs on the storage nodes and the only showing in them is that it is not the master node.

[Sebastian Roth]

@Obsidian So do you still need help on this? If so, please provide more information like log files.

[Wayne Workman]

/opt/fog/log/fogreplicator.log I think. Last 100 or so lines should be enough.

[Obsidian]

This is from one of the storage nodes:

 vi /opt/fog/log/fogreplicator.log
[01-15-18 2:39:05 am]  *  | This is not the master node
[01-15-18 2:49:05 am]  *  | This is not the master node
[01-15-18 2:59:05 am]  *  | This is not the master node
[01-15-18 3:09:05 am]  *  | This is not the master node
[01-15-18 3:19:05 am]  *  | This is not the master node
[01-15-18 3:29:05 am]  *  | This is not the master node
[01-15-18 3:39:05 am]  *  | This is not the master node
[01-15-18 3:49:05 am]  *  | This is not the master node
[01-15-18 3:59:05 am]  *  | This is not the master node
[01-15-18 4:09:05 am]  *  | This is not the master node
[01-15-18 4:19:05 am]  *  | This is not the master node
[01-15-18 4:29:05 am]  *  | This is not the master node
[01-15-18 4:39:05 am]  *  | This is not the master node
[01-15-18 4:49:05 am]  *  | This is not the master node
[01-15-18 4:59:05 am]  *  | This is not the master node
[01-15-18 5:09:05 am]  *  | This is not the master node
[01-15-18 5:19:05 am]  *  | This is not the master node
[01-15-18 5:29:06 am]  *  | This is not the master node
[01-15-18 5:39:06 am]  *  | This is not the master node
[01-15-18 5:49:06 am]  *  | This is not the master node
[01-15-18 5:59:06 am]  *  | This is not the master node
[01-15-18 6:09:06 am]  *  | This is not the master node
[01-15-18 6:19:06 am]  *  | This is not the master node
[01-15-18 6:29:06 am]  *  | This is not the master node
[01-15-18 6:39:06 am]  *  | This is not the master node
[01-15-18 6:49:06 am]  *  | This is not the master node
[01-15-18 6:59:06 am]  *  | This is not the master node
[01-15-18 7:09:06 am]  *  | This is not the master node
[01-15-18 7:19:06 am]  *  | This is not the master node
[01-15-18 7:29:06 am]  *  | This is not the master node
[01-15-18 7:39:06 am]  *  | This is not the master node
[01-15-18 7:49:06 am]  *  | This is not the master node
[01-15-18 7:59:06 am]  *  | This is not the master node
[01-15-18 8:09:06 am]  *  | This is not the master node
[01-15-18 8:19:07 am]  *  | This is not the master node
[01-15-18 8:29:07 am]  *  | This is not the master node
[01-15-18 8:39:07 am]  *  | This is not the master node
[01-15-18 8:49:07 am]  *  | This is not the master node
[01-15-18 8:59:07 am]  *  | This is not the master node
[01-15-18 9:09:07 am]  *  | This is not the master node
[01-15-18 9:19:07 am]  *  | This is not the master node
[01-15-18 9:29:07 am]  *  | This is not the master node
[01-15-18 9:39:07 am]  *  | This is not the master node
[01-15-18 9:49:07 am]  *  | This is not the master node
[01-15-18 9:59:07 am]  *  | This is not the master node
[01-15-18 10:09:07 am]  *  | This is not the master node
[01-15-18 10:19:07 am]  *  | This is not the master node
[01-15-18 10:29:07 am]  *  | This is not the master node
[01-15-18 10:39:07 am]  *  | This is not the master node
[01-15-18 10:49:07 am]  *  | This is not the master node
[01-15-18 10:59:07 am]  *  | This is not the master node
[01-15-18 11:09:07 am]  *  | This is not the master node
[01-15-18 11:19:07 am]  *  | This is not the master node
[01-15-18 11:29:07 am]  *  | This is not the master node
[01-15-18 11:39:07 am]  *  | This is not the master node
[01-15-18 11:49:08 am]  *  | This is not the master node
[01-15-18 11:59:08 am]  *  | This is not the master node
[01-15-18 12:09:08 pm]  *  | This is not the master node
[01-15-18 12:19:08 pm]  *  | This is not the master node
[01-15-18 12:29:08 pm]  *  | This is not the master node
[01-15-18 12:39:08 pm]  *  | This is not the master node
[01-15-18 12:49:08 pm]  *  | This is not the master node
[01-15-18 12:59:08 pm]  *  | This is not the master node

Here is for the Master

[01-15-18 7:26:43 pm]  * Started sync for Image postdownloadscripts
[01-15-18 7:26:44 pm]  | postdownloadscripts: No need to sync fog.postdownload file to Frankfort
[01-15-18 7:26:44 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.Frankfort.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:3840000;set net:limit-rate 0:512000; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.13.1.16
[01-15-18 7:26:44 pm]  * Started sync for Image postdownloadscripts
[01-15-18 7:26:44 pm] | Replication already running with PID: 3885
[01-15-18 7:26:48 pm]  * Type: 2, File: /var/www/html/fog/lib/fog/fogftp.class.php, Line: 463, Message: ftp_login(): Login incorrect., Host: 10.7.1.16, Username: fog
[01-15-18 7:36:39 pm]  * Starting Image Replication.
[01-15-18 7:36:39 pm]  * We are group ID: 1. We are group name: default
[01-15-18 7:36:40 pm]  * We are node ID: 1. We are node name: Summit
[01-15-18 7:36:40 pm]  * Attempting to perform Group -> Group image replication.
[01-15-18 7:36:40 pm]  | Replicating postdownloadscripts
[01-15-18 7:36:40 pm]  * Found Image to transfer to 11 nodes
[01-15-18 7:36:40 pm]  | File Name: postdownloadscripts
[01-15-18 7:36:40 pm]  | 236 0 /images/postdownloadscripts/fog.postdownload
[01-15-18 7:36:40 pm]  | Files do not match.
[01-15-18 7:36:40 pm]  * Deleting remote file: /images/postdownloadscripts/
[01-15-18 7:36:41 pm]  * Starting Sync Actions
[01-15-18 7:36:41 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.Alsip.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:3840000;set net:limit-rate 0:512000; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.6.1.16
[01-15-18 7:36:41 pm]  * Started sync for Image postdownloadscripts
[01-15-18 7:36:42 pm]  | postdownloadscripts: No need to sync fog.postdownload file to Byron Center
[01-15-18 7:36:42 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.Byron Center.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:3840000;set net:limit-rate 0:512000; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.1.1.16
[01-15-18 7:36:42 pm]  * Started sync for Image postdownloadscripts
[01-15-18 7:36:44 pm]  | postdownloadscripts: No need to sync fog.postdownload file to Frankfort
[01-15-18 7:36:44 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.Frankfort.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:3840000;set net:limit-rate 0:512000; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.13.1.16
[01-15-18 7:36:44 pm]  * Started sync for Image postdownloadscripts
[01-15-18 7:36:44 pm] | Replication already running with PID: 3885
[01-15-18 7:36:47 pm]  * Type: 2, File: /var/www/html/fog/lib/fog/fogftp.class.php, Line: 463, Message: ftp_login(): Login incorrect., Host: 10.7.1.16, Username: fog
[01-15-18 7:46:40 pm]  * Starting Image Replication.
[01-15-18 7:46:40 pm]  * We are group ID: 1. We are group name: default
[01-15-18 7:46:40 pm]  * We are node ID: 1. We are node name: Summit
[01-15-18 7:46:40 pm]  * Attempting to perform Group -> Group image replication.
[01-15-18 7:46:40 pm]  | Replicating postdownloadscripts
[01-15-18 7:46:40 pm]  * Found Image to transfer to 11 nodes
[01-15-18 7:46:40 pm]  | File Name: postdownloadscripts
[01-15-18 7:46:41 pm]  | 236 0 /images/postdownloadscripts/fog.postdownload
[01-15-18 7:46:41 pm]  | Files do not match.
[01-15-18 7:46:41 pm]  * Deleting remote file: /images/postdownloadscripts/
[01-15-18 7:46:41 pm]  * Starting Sync Actions
[01-15-18 7:46:41 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.Alsip.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:3840000;set net:limit-rate 0:512000; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.6.1.16
[01-15-18 7:46:41 pm]  * Started sync for Image postdownloadscripts
[01-15-18 7:46:42 pm]  | postdownloadscripts: No need to sync fog.postdownload file to Byron Center
[01-15-18 7:46:42 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.Byron Center.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:3840000;set net:limit-rate 0:512000; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.1.1.16
[01-15-18 7:46:42 pm]  * Started sync for Image postdownloadscripts
[01-15-18 7:46:44 pm]  | postdownloadscripts: No need to sync fog.postdownload file to Frankfort
[01-15-18 7:46:44 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.Frankfort.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:3840000;set net:limit-rate 0:512000; mirror -c -r -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.13.1.16
[01-15-18 7:46:44 pm]  * Started sync for Image postdownloadscripts
[01-15-18 7:46:44 pm] | Replication already running with PID: 3885
[01-15-18 7:46:48 pm]  * Type: 2, File: /var/www/html/fog/lib/fog/fogftp.class.php, Line: 463, Message: ftp_login(): Login incorrect., Host: 10.7.1.16, Username: fog

[Wayne Workman]

@obsidian said in Fog Replication Not working:

Message: ftp_login(): Login incorrect., Host: 10.7.1.16, Username: fog

That’s the problem. Doublecheck your passwords. Make sure the password is set correctly inside each storage node’s /opt/fog/.fogsettings file. Re-run the installer on those to correct the problem. The installer will update the node’s credentials in the database too (or should anyway).

[Obsidian]

I will do that today. Would that stop the synchronization of the ones that are correct or will it only just not synchronize the one that is wrong and continue updating the ones that are correct?

[Wayne Workman]

@obsidian The ones that are correct stay correct.

[Obsidian]

That is what I thought. So I am confused as to why the Windows 10 image has not replicated to the ones that are syncing.

[Wayne Workman]

@obsidian said in Fog Replication Not working:

So I am confused as to why the Windows 10 image has not replicated to the ones that are syncing.

I have no idea. Check the master node’s replication log. It should give a reason.