Replication: Only working on 1/5 nodes

[dburk]

I have Fog setup on 6 Ubuntu Servers.
They are configured as follows:

For some reason replication is only working on CHS-FOG, however its continuously deleting and uploading some of the files.
The rest of my storage nodes are not receiving anything.

Replication Log:

lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.CHS-FOG.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.2.31.16
[06-07-18 12:45:48 pm]  * Started sync for Image postdownloadscripts
[06-07-18 12:45:48 pm] | Replication already running with PID: 7746
[06-07-18 12:45:48 pm] | Replication already running with PID: 7746
[06-07-18 12:45:48 pm] | Replication already running with PID: 7746
[06-07-18 12:45:48 pm] | Replication already running with PID: 7746
[06-07-18 12:45:48 pm]  | Replicating postinitscripts
[06-07-18 12:45:50 pm]  * Found Image to transfer to 5 s
[06-07-18 12:45:50 pm]  | File Name: dev/postinitscripts
[06-07-18 12:45:51 pm]  | dev/postinitscripts: No need to sync fog.postinit file to CHS-FOG
[06-07-18 12:45:51 pm]  * All files synced for this item.
[06-07-18 12:45:52 pm]  | dev/postinitscripts: No need to sync fog.postinit file to CMS-FOG
[06-07-18 12:45:52 pm]  * All files synced for this item.
[06-07-18 12:45:53 pm]  | dev/postinitscripts: No need to sync fog.postinit file to EST-FOG
[06-07-18 12:45:53 pm]  * All files synced for this item.
[06-07-18 12:45:54 pm]  | dev/postinitscripts: No need to sync fog.postinit file to MAG-FOG
[06-07-18 12:45:54 pm]  * All files synced for this item.
[06-07-18 12:45:55 pm]  | dev/postinitscripts: No need to sync fog.postinit file to WRP-FOG
[06-07-18 12:45:55 pm]  * All files synced for this item.
[06-07-18 12:45:55 pm]  * Not syncing Image between s
[06-07-18 12:45:55 pm]  | Image Name: w7-32bd
[06-07-18 12:45:55 pm]  | There are no other members to sync to.
[06-07-18 12:45:56 pm]  * Not syncing Image between s
[06-07-18 12:45:56 pm]  | Image Name: w7-64bd
[06-07-18 12:45:56 pm]  | There are no other members to sync to.
[06-07-18 12:45:56 pm]  * Attempting to perform Group -> Nodes image replication.
[06-07-18 12:45:58 pm]  * Found Image to transfer to 5 s
[06-07-18 12:45:58 pm]  | Image Name: w7-32bd
[06-07-18 12:45:59 pm]  | w7-32bd: No need to sync d1.fixed_size_partitions file to CHS-FOG
[06-07-18 12:45:59 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:45:59 pm]  | Deleting remote file: /images/w7-32bd/d1.mbr
[06-07-18 12:46:00 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:46:00 pm]  | Deleting remote file: /images/w7-32bd/d1.minimum.partitions
[06-07-18 12:46:00 pm]  | w7-32bd: No need to sync d1.original.fstypes file to CHS-FOG
[06-07-18 12:46:00 pm]  | w7-32bd: No need to sync d1.original.swapuuids file to CHS-FOG
[06-07-18 12:46:00 pm]  | w7-32bd: No need to sync d1.partitions file to CHS-FOG
[06-07-18 12:49:00 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:49:00 pm]  | Deleting remote file: /images/w7-32bd/d1p1.img
[06-07-18 12:49:00 pm]  * Starting Sync Actions
[06-07-18 12:49:00 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.w7-32bd.transfer.CHS-FOG.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/w7-32bd" "/images/w7-32bd"; exit' -u fog,[Protected] 10.2.31.16
[06-07-18 12:49:00 pm]  * Started sync for Image w7-32bd
[06-07-18 12:49:00 pm] | Replication already running with PID: 8463
[06-07-18 12:49:00 pm] | Replication already running with PID: 8463
[06-07-18 12:49:00 pm] | Replication already running with PID: 8463
[06-07-18 12:49:00 pm] | Replication already running with PID: 8463
[06-07-18 12:49:01 pm]  * Found Image to transfer to 5 s
[06-07-18 12:49:01 pm]  | Image Name: w7-64bd
[06-07-18 12:49:02 pm]  | w7-64bd: No need to sync d1.fixed_size_partitions file to CHS-FOG
[06-07-18 12:49:02 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:49:02 pm]  | Deleting remote file: /images/w7-64bd/d1.mbr
[06-07-18 12:49:03 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:49:03 pm]  | Deleting remote file: /images/w7-64bd/d1.minimum.partitions
[06-07-18 12:49:03 pm]  | w7-64bd: No need to sync d1.original.fstypes file to CHS-FOG
[06-07-18 12:49:03 pm]  | w7-64bd: No need to sync d1.original.swapuuids file to CHS-FOG
[06-07-18 12:49:04 pm]  | w7-64bd: No need to sync d1.partitions file to CHS-FOG
[06-07-18 12:53:13 pm]  | w7-64bd: No need to sync d1p1.img file to CHS-FOG
[06-07-18 12:53:13 pm]  * Starting Sync Actions
[06-07-18 12:53:13 pm]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.w7-64bd.transfer.CHS-FOG.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/w7-64bd" "/images/w7-64bd"; exit' -u fog,[Protected] 10.2.31.16
[06-07-18 12:53:13 pm]  * Started sync for Image w7-64bd
[06-07-18 12:53:13 pm] | Replication already running with PID: 9184
[06-07-18 12:53:13 pm] | Replication already running with PID: 9184
[06-07-18 12:53:13 pm] | Replication already running with PID: 9184
[06-07-18 12:53:13 pm] | Replication already running with PID: 9184
[06-07-18 12:53:14 pm]  * Starting Image Replication.
[06-07-18 12:53:14 pm]  * We are group ID: 1. We are group name: MAIN
[06-07-18 12:53:14 pm]  * We are node ID: 1. We are node name: Fog-Main
[06-07-18 12:53:14 pm]  * Attempting to perform Group -> Group image replication.
[06-07-18 12:53:14 pm]  | Replicating postdownloadscripts
[06-07-18 12:53:15 pm]  * Found Image to transfer to 5 s
[06-07-18 12:53:15 pm]  | File Name: postdownloadscripts
[06-07-18 12:53:16 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:53:16 pm]  | Deleting remote file: /images/postdownloadscripts/devScripts/7zToGZ.sh
[06-07-18 12:53:16 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:53:16 pm]  | Deleting remote file: /images/postdownloadscripts/devScripts/7zToGZ.sh
[06-07-18 12:53:16 pm]  | postdownloadscripts: No need to sync getModel.sh file to CHS-FOG
[06-07-18 12:53:21 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:53:21 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/20BG0011US.tar.gz
[06-07-18 12:53:25 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:53:25 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/20BG0011US.tar.gz
[06-07-18 12:53:29 pm]  | Files do not match on server: CHS-FOG
[06-07-18 12:53:29 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/20BG0011US.tar.gz
[06-07-18 12:53:41 pm]  | postdownloadscripts: No need to sync 20BG0011US.tar.gz file to CHS-FOG
[06-07-18 12:53:57 pm]  | postdownloadscripts: No need to sync 2344BNU.tar.gz file to CHS-FOG
[06-07-18 12:54:12 pm]  | postdownloadscripts: No need to sync 2344BZU.tar.gz file to CHS-FOG
[06-07-18 12:54:28 pm]  | postdownloadscripts: No need to sync 2349DE6.tar.gz file to CHS-FOG
[06-07-18 12:54:44 pm]  | postdownloadscripts: No need to sync 2349NL5.tar.gz file to CHS-FOG
[06-07-18 12:55:09 pm]  | postdownloadscripts: No need to sync 3238BL1.tar.gz file to CHS-FOG
[06-07-18 12:55:15 pm]  | postdownloadscripts: No need to sync 3340.tar.gz file to CHS-FOG
[06-07-18 12:55:31 pm]  | postdownloadscripts: No need to sync 33722FU.tar.gz file to CHS-FOG
[06-07-18 12:55:47 pm]  | postdownloadscripts: No need to sync 33722VU.tar.gz file to CHS-FOG
[06-07-18 12:56:35 pm]  | postdownloadscripts: No need to sync 4236S3G.tar.gz file to CHS-FOG
[06-07-18 12:56:42 pm]  | postdownloadscripts: No need to sync 7010.tar.gz file to CHS-FOG
[06-07-18 12:56:47 pm]  | postdownloadscripts: No need to sync 760.tar.gz file to CHS-FOG
[06-07-18 12:57:15 pm]  | postdownloadscripts: No need to sync 7720.tar.gz file to CHS-FOG
[06-07-18 12:57:24 pm]  | postdownloadscripts: No need to sync 780.tar.gz file to CHS-FOG
[06-07-18 12:57:30 pm]  | postdownloadscripts: No need to sync 790.tar.gz file to CHS-FOG
[06-07-18 12:57:40 pm]  | postdownloadscripts: No need to sync 9010.tar.gz file to CHS-FOG
[06-07-18 12:57:52 pm]  | postdownloadscripts: No need to sync 990.tar.gz file to CHS-FOG
[06-07-18 12:58:00 pm]  | postdownloadscripts: No need to sync E5430.tar.gz file to CHS-FOG
[06-07-18 12:58:13 pm]  | postdownloadscripts: No need to sync E5440.tar.gz file to CHS-FOG
[06-07-18 12:58:19 pm]  | postdownloadscripts: No need to sync E5510.tar.gz file to CHS-FOG
[06-07-18 12:58:22 pm]  | postdownloadscripts: No need to sync E5520.tar.gz file to CHS-FOG
[06-07-18 12:58:29 pm]  | postdownloadscripts: No need to sync E5530.tar.gz file to CHS-FOG
[06-07-18 12:58:41 pm]  | postdownloadscripts: No need to sync E5540.tar.gz file to CHS-FOG
[06-07-18 12:58:50 pm]  | postdownloadscripts: No need to sync E5550.tar.gz file to CHS-FOG
[06-07-18 12:59:03 pm]  | postdownloadscripts: No need to sync E5570.tar.gz file to CHS-FOG
[06-07-18 12:59:09 pm]  | postdownloadscripts: No need to sync E6400.tar.gz file to CHS-FOG
[06-07-18 12:59:19 pm]  | postdownloadscripts: No need to sync E6410.tar.gz file to CHS-FOG
[06-07-18 12:59:27 pm]  | postdownloadscripts: No need to sync E6420.tar.gz file to CHS-FOG
[06-07-18 12:59:32 pm]  | postdownloadscripts: No need to sync E6500.tar.gz file to CHS-FOG
[06-07-18 12:59:40 pm]  | postdownloadscripts: No need to sync E6510.tar.gz file to CHS-FOG
[06-07-18 12:59:44 pm]  | postdownloadscripts: No need to sync M4500.tar.gz file to CHS-FOG
[06-07-18 12:59:49 pm]  | postdownloadscripts: No need to sync M4600.tar.gz file to CHS-FOG
[06-07-18 1:00:04 pm]  | postdownloadscripts: No need to sync M6800.tar.gz file to CHS-FOG
[06-07-18 1:00:17 pm]  | postdownloadscripts: No need to sync T1700.tar.gz file to CHS-FOG
[06-07-18 1:00:23 pm]  | postdownloadscripts: No need to sync T3500.tar.gz file to CHS-FOG
[06-07-18 1:01:27 pm]  | postdownloadscripts: No need to sync T420.tar.gz file to CHS-FOG
[06-07-18 1:01:40 pm]  | postdownloadscripts: No need to sync T430.tar.gz file to CHS-FOG
[06-07-18 1:01:46 pm]  | postdownloadscripts: No need to sync T5500.tar.gz file to CHS-FOG
[06-07-18 1:02:01 pm]  | postdownloadscripts: No need to sync W540.tar.gz file to CHS-FOG
[06-07-18 1:02:18 pm]  | postdownloadscripts: No need to sync X131e.tar.gz file to CHS-FOG
[06-07-18 1:02:58 pm]  | postdownloadscripts: No need to sync x100e.tar.gz file to CHS-FOG
[06-07-18 1:02:59 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:02:59 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x86/1501.tar.gz
[06-07-18 1:03:03 pm]  | postdownloadscripts: No need to sync 1501.tar.gz file to CHS-FOG
[06-07-18 1:03:06 pm]  | postdownloadscripts: No need to sync 2100.tar.gz file to CHS-FOG
[06-07-18 1:03:07 pm]  | postdownloadscripts: No need to sync 2110.tar.gz file to CHS-FOG
[06-07-18 1:03:10 pm]  | postdownloadscripts: No need to sync 2120.tar.gz file to CHS-FOG
[06-07-18 1:03:15 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:03:15 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/2344BNU.tar.gz
[06-07-18 1:03:21 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:03:21 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/2344BZU.tar.gz
[06-07-18 1:03:25 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:03:25 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/2349DE6.tar.gz
[06-07-18 1:03:31 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:03:31 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/2349NL5.tar.gz
[06-07-18 1:03:43 pm]  | postdownloadscripts: No need to sync 2349S4R.tar.gz file to CHS-FOG
[06-07-18 1:03:45 pm]  | postdownloadscripts: No need to sync 28762JU.tar.gz file to CHS-FOG
[06-07-18 1:04:00 pm]  | postdownloadscripts: No need to sync 311-1000.tar.gz file to CHS-FOG
[06-07-18 1:04:09 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:09 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/3238BL1.tar.gz
[06-07-18 1:04:11 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:11 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/3340.tar.gz
[06-07-18 1:04:17 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:17 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/33722FU.tar.gz
[06-07-18 1:04:22 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:22 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/33722VU.tar.gz
[06-07-18 1:04:24 pm]  | postdownloadscripts: No need to sync 350829U.tar.gz file to CHS-FOG
[06-07-18 1:04:26 pm]  | postdownloadscripts: No need to sync 390.tar.gz file to CHS-FOG
[06-07-18 1:04:42 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:42 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/4236S3G.tar.gz
[06-07-18 1:04:44 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:44 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/7010.tar.gz
[06-07-18 1:04:45 pm]  | postdownloadscripts: No need to sync 740.tar.gz file to CHS-FOG
[06-07-18 1:04:46 pm]  | postdownloadscripts: No need to sync 745.tar.gz file to CHS-FOG
[06-07-18 1:04:47 pm]  | postdownloadscripts: No need to sync 755.tar.gz file to CHS-FOG
[06-07-18 1:04:48 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:49 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/760.tar.gz
[06-07-18 1:04:51 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:51 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/790.tar.gz
[06-07-18 1:04:54 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:54 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/9010.tar.gz
[06-07-18 1:04:57 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:04:57 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/990.tar.gz
[06-07-18 1:04:59 pm]  | postdownloadscripts: No need to sync D620.tar.gz file to CHS-FOG
[06-07-18 1:05:00 pm]  | postdownloadscripts: No need to sync D630.tar.gz file to CHS-FOG
[06-07-18 1:05:08 pm]  | postdownloadscripts: No need to sync DM1.tar.gz file to CHS-FOG
[06-07-18 1:05:10 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:10 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E5430.tar.gz
[06-07-18 1:05:14 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:14 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E5440.tar.gz
[06-07-18 1:05:16 pm]  | postdownloadscripts: No need to sync E5500.tar.gz file to CHS-FOG
[06-07-18 1:05:18 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:18 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E5510.tar.gz
[06-07-18 1:05:19 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:19 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E5520.tar.gz
[06-07-18 1:05:21 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:21 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E5530.tar.gz
[06-07-18 1:05:25 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:25 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E5540.tar.gz
[06-07-18 1:05:28 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:28 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E5550.tar.gz
[06-07-18 1:05:32 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:32 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E5570.tar.gz
[06-07-18 1:05:34 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:34 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E6400.tar.gz
[06-07-18 1:05:36 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:36 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E6410.tar.gz
[06-07-18 1:05:39 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:39 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E6420.tar.gz
[06-07-18 1:05:46 pm]  | postdownloadscripts: No need to sync E6430.tar.gz file to CHS-FOG
[06-07-18 1:05:48 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:48 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E6500.tar.gz
[06-07-18 1:05:50 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:50 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/E6510.tar.gz
[06-07-18 1:05:53 pm]  | postdownloadscripts: No need to sync E6520.tar.gz file to CHS-FOG
[06-07-18 1:05:54 pm]  | postdownloadscripts: No need to sync GX270.tar.gz file to CHS-FOG
[06-07-18 1:05:54 pm]  | postdownloadscripts: No need to sync GX280.tar.gz file to CHS-FOG
[06-07-18 1:05:54 pm]  | postdownloadscripts: No need to sync GX520.tar.gz file to CHS-FOG
[06-07-18 1:05:59 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:05:59 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/M6800.tar.gz
[06-07-18 1:06:03 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:06:03 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/T1700.tar.gz
[06-07-18 1:06:04 pm]  | postdownloadscripts: No need to sync T3400.tar.gz file to CHS-FOG
[06-07-18 1:06:06 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:06:06 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/T3500.tar.gz
[06-07-18 1:06:24 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:06:24 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/T420.tar.gz
[06-07-18 1:06:28 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:06:28 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/T430.tar.gz
[06-07-18 1:06:30 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:06:30 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/T5500.tar.gz
[06-07-18 1:06:34 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:06:34 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/W540.tar.gz
[06-07-18 1:06:39 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:06:39 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/X131e.tar.gz
[06-07-18 1:06:51 pm]  | Files do not match on server: CHS-FOG
[06-07-18 1:06:51 pm]  | Deleting remote file: /images/postdownloadscripts/drivers/W7/x64/x100e.tar.gz
[06-07-18 1:06:52 pm]  | postdownloadscripts: No need to sync fog.injectDrivers file to CHS-FOG
[06-07-18 1:06:52 pm]  | postdownloadscripts: No need to sync fog.injectDrivers.save file to CHS-FOG
[06-07-18 1:06:52 pm]  | postdownloadscripts: No need to sync fog.postdownload file to CHS-FOG```

[jflippen]

@dburk Hey, sounds like you’re having a similar issue to me. I’ve been posting on this thread here:
https://forums.fogproject.org/topic/11856/fog-storage-node-ftp-error-in-image-replicator-log-files-do-not-match/9

From what I’ve found so far, it seems that FOG is able to create the initial upload just fine, but is not deleting files that need to be changed after making updates to an image. In relation to your specific issue, I have also noticed during my troubleshooting that FOG will only try to update one node at a time per image. Since it keeps failing to properly delete and transfer the new file versions of your image to CHS-FOG, it is stuck in a loop and keeps trying to update the files over and over again to the same node.

Is the image being replicated for the first time or is this an update to an image that has already transferred once? If it is the later and you are looking for a quick fix, you could delete the image from your /images folder in each node and let the replication service send over a fresh copy of the image. This has worked for me until I can figure out why the replication service won’t update existing images on nodes. I am also toying around with rsync commands to manually replicate my images out to each node, since I do that anyways for my drivers folder used to driver injection based off this post:
https://forums.fogproject.org/topic/11126/using-fog-postinstall-scripts-for-windows-driver-injection-2017-ed

I am going to keep following your post to see what other people say, as I think it is related to the problem I am having as well.

[dburk]

Just deleted the image files from my CHS node. Looks like replication is starting to do its job again.

In regards to drivers I’m not sure if this is best practice, but i have my drivers living in the /images/postdownloadscripts/drivers directory so that they also get replicated.

I’ll Keep you posted. Thanks for the Info.

[george1421]

@jflippen said in Replication: Only working on 1/5 nodes:

This has worked for me until I can figure out why the replication service won’t update existing images on nodes. I am also toying around with rsync commands to manually replicate my images out to each node, since I do that anyways for my drivers folder used to driver injection based off this post:
https://forums.fogproject.org/topic/11126/using-fog-postinstall-scripts-for-windows-driver-injection-2017-ed

You can get the replicator to sync your drivers directory if you create a faux image definition in FOG called Drivers and then make it reference the directory /images/drivers there is no need to run a separate rsync process.

In regards to the FOG Image replicator, it has been changed over the last few releases to only support FOG storage nodes. If you have a NAS as a storage node, image replication will keep cycling over and over again. It will happen but it will be continuous. If you have this type of environment then setting up a cron job to run the rsync process is a pretty solid one. Many nas’ have an rsync process that you can enable to make the replication process a bit quicker and more efficient. I would eventually like to see FOG switch to an rsync process and move away from the custom code to offer a bit more flexibility.

[jflippen]

@george1421 Hey George, I thought the faux drivers folder would only work if there were no subdirectories in the drivers folder? Has this been modified to allow your previous setup of drivers/ModelName/x64/.ini files?

Also, I would like to note that I am not using NAS devices but three Dell rack mounted servers that we bought a few months ago. I do have a hardware-level RAID 1, but today I have repeated the test and set up some old OptPlex 380 machines as test fog servers (as their own master and 2 nodes, not as part of my actual server and ther nodes). With these test fog servers, it’s even more screwy. It says that files don’t match even though the md5sum is the same. I didn’t even get to updating the image. The imagereplicationservice is now hanging up after the first transfer (on the 380’s at least).

I was trying to find the code to get a better understanding of what’s happening at the actual service level. After some digging I found the imagereplicator.class.php file, which calls another function replicateItems for each image. I cannot find where this replicateItems function is defined. Basically, I am trying to find the class that is actually doing the lftp command and / or deleting the files, how it’s verifying if they are the same, etc.

You wouldn’t happen to know where these files are, would you?

[george1421]

@jflippen The last I knew the entire content of the /image/drivers directory is replicated. I do have to say we don’t change drivers much and something may have change since all of the drivers are on all storage nodes at the moment. I’ll have to confirm it still works as I thought.

both the image as well as the snapin replicator should use the lftp command defined in this file fog/lib/service/fogservice.class.php

Double checking I don’t see any other instance of lftp.

[jflippen]

@george1421 Thanks, George. I’ll have to give that a go once I get this replication issue resolved. Right now I’m trying to keep myself from going cross-eyed going over @Tom-Elliott 's PHP. It’s driving me nuts that I can’t figure out:

• Why it says the file is different when sometimes the md5sum matches for the two files

• Why it’s not deleting the file when they don’t match

• When replacing a file with a different md5sum, if I run the lftp command manually and insert the correct credentials, I don’t get any errors, but it doesn’t change the existing file (I’m guessing since the deleting is handled by this line: self::$FOGFTP->delete($remotefilescheck[$index]); )

To be fair, I don’t know PHP, just some other languages. I am just trying to get a better understanding of what’s happening under the hood to see if it will help me resolve my issue.

I feel like there has to be something wrong with my environment, otherwise we should see way more people complaining about this issue, no? However the fact I can replicate the issue on the OplixPlex 380 test fog group with the issues I am having with my actual fog servers means it’s likely not the servers themselves…or I’m doing it wrong after following the directions here:
https://wiki.fogproject.org/wiki/index.php?title=CentOS_7

Idk… I better start sweeping up all the hair I’ve pulled out

[jflippen]

@george1421 @Tom-Elliott Is this a logic error in this method?

 private static function _filesAreEqual($size_a, $size_b, $file_a, $file_b, $avail) {
        if ($size_a != $size_b) {
            return false;
        }
        if (false === $avail) {
            if ($size_a < 1047685760) {
                $remhash = md5_file($file_b);
                $lochash = md5_file($file_a);
                return ($remhash == $lochash);
            }
            return file_exists($file_b) && file_exists($file_a);
        }
        $hashLoc = self::getHash($file_a);
        $hashRem = $file_b;
        $hashCom = ($hashLoc == $hashRem);
        return $hashCom;

It looks like it skips md5sum for files larger than 1GB and does the encode_64 from getHash instead. The method is calling getHash for $file_a but not for $file_b. I do not see a part in the php file that calls for a hash for $file_b at all. Wouldn’t this mean the hashComp would be comparing a hashed file vs a non-hashed file?

[Tom Elliott]

@jflippen file b is handled separately and done on the remote side. This method receives that hash and uses it

[jflippen]

@tom-elliott Okay, thanks. Back to the drawing board I go. Any ideas on why this line, self::$FOGFTP->delete($remotefilescheck[$index]); , is not deleting the file on the node before the lftp call?

[george1421]

@jflippen Each fog server has a local user account called fog with its own password. Have you tried to log into the remote fog server using the appropriate linux user fog password as defined in the storage node configuration for that storage node and done by ftp?

I know that was a mouth full, but the basic premise is to ftp to the remote storage node using the appropriate fog account. Change to the /images directory and ensure that you can put and delete files over ftp.

[jflippen]

@george1421 Sorry for the long delay, I’ve been gone on vacation for the last week. I just tried using the fog credentials stored for one of the nodes and was able to FTP in from my Windows workstation using clonezilla. I was able to delete the file manually. So now the question is:
Why isn’t the PHP command to do the FTP deletion for the replicator service working?

[Sebastian Roth]

@dburk Sorry for digging out this old topic. While trying to figure out some replication hickup I found and fixed a couple of issues within the code. All that will be in the next release. Hopefully coming soon. Let me know if you are keen to test those changes beforehand.

Edit: Issues fixed in 1.5.5.