Replication problems 1.5.4 - always copying

JGallo

@jflippen I to updated and noticed it is no longer in a replication loop. Interesting as I observed the replication log go through all my storage nodes is that first storage node has same file that does not match and one other storage node. So i’m waiting for next round of replication to occur and continue to observe. Update seem to fix the processes that @Tom-Elliott was talking about earlier and seems to check other storage nodes as replication goes down the line of storage nodes.

JGallo

I got a undefined variable error in replication log at the bottom. As you can see, replication is working but still states that d1p2.img file doesn’t match. Also, the file for the storage node with the error was perfectly fine on an earlier check. I didn’t update any images today.

[07-09-18 5:51:14 am]  | There are no other members to sync to.
[07-09-18 5:51:17 am]  * Found Image to transfer to 8 s
[07-09-18 5:51:17 am]  | Image Name: 32-Dell-790
[07-09-18 5:51:18 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to BCS-Master
[07-09-18 5:51:18 am]  | 32-Dell-790: No need to sync d1.mbr file to BCS-Master
[07-09-18 5:51:19 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to BCS-Master
[07-09-18 5:51:20 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to BCS-Master
[07-09-18 5:51:21 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to BCS-Master
[07-09-18 5:51:21 am]  | 32-Dell-790: No need to sync d1.partitions file to BCS-Master
[07-09-18 5:51:23 am]  | 32-Dell-790: No need to sync d1p1.img file to BCS-Master
[07-09-18 5:58:23 am]  | 32-Dell-790: No need to sync d1p2.img file to BCS-Master
[07-09-18 5:58:23 am]  * All files synced for this item.
[07-09-18 5:58:24 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to CHS-Master
[07-09-18 5:58:24 am]  | 32-Dell-790: No need to sync d1.mbr file to CHS-Master
[07-09-18 5:58:25 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to CHS-Master
[07-09-18 5:58:25 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to CHS-Master
[07-09-18 5:58:26 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to CHS-Master
[07-09-18 5:58:26 am]  | 32-Dell-790: No need to sync d1.partitions file to CHS-Master
[07-09-18 5:58:26 am]  | 32-Dell-790: No need to sync d1p1.img file to CHS-Master
[07-09-18 6:03:39 am]  | 32-Dell-790: No need to sync d1p2.img file to CHS-Master
[07-09-18 6:03:39 am]  * All files synced for this item.
[07-09-18 6:03:39 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to ESC-Master
[07-09-18 6:03:40 am]  | 32-Dell-790: No need to sync d1.mbr file to ESC-Master
[07-09-18 6:03:40 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to ESC-Master
[07-09-18 6:03:40 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to ESC-Master
[07-09-18 6:03:41 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to ESC-Master
[07-09-18 6:03:41 am]  | 32-Dell-790: No need to sync d1.partitions file to ESC-Master
[07-09-18 6:03:41 am]  | 32-Dell-790: No need to sync d1p1.img file to ESC-Master
[07-09-18 6:07:44 am]  | 32-Dell-790: No need to sync d1p2.img file to ESC-Master
[07-09-18 6:07:44 am]  * All files synced for this item.
[07-09-18 6:07:45 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to Jep-Master
[07-09-18 6:07:45 am]  | 32-Dell-790: No need to sync d1.mbr file to Jep-Master
[07-09-18 6:07:46 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to Jep-Master
[07-09-18 6:07:46 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to Jep-Master
[07-09-18 6:07:47 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to Jep-Master
[07-09-18 6:07:47 am]  | 32-Dell-790: No need to sync d1.partitions file to Jep-Master
[07-09-18 6:07:48 am]  | 32-Dell-790: No need to sync d1p1.img file to Jep-Master
[07-09-18 6:13:05 am]  | 32-Dell-790: No need to sync d1p2.img file to Jep-Master
[07-09-18 6:13:05 am]  * All files synced for this item.
[07-09-18 6:13:06 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VHS-Master
[07-09-18 6:13:07 am]  | 32-Dell-790: No need to sync d1.mbr file to VHS-Master
[07-09-18 6:13:07 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to VHS-Master
[07-09-18 6:13:08 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to VHS-Master
[07-09-18 6:13:08 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to VHS-Master
[07-09-18 6:13:09 am]  | 32-Dell-790: No need to sync d1.partitions file to VHS-Master
[07-09-18 6:13:10 am]  | 32-Dell-790: No need to sync d1p1.img file to VHS-Master
[07-09-18 6:19:08 am]  | 32-Dell-790: No need to sync d1p2.img file to VHS-Master
[07-09-18 6:19:08 am]  * All files synced for this item.
[07-09-18 6:19:10 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VP-Master
[07-09-18 6:19:11 am]  | 32-Dell-790: No need to sync d1.mbr file to VP-Master
[07-09-18 6:19:11 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to VP-Master
[07-09-18 6:19:12 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to VP-Master
[07-09-18 6:19:13 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to VP-Master
[07-09-18 6:19:13 am]  | 32-Dell-790: No need to sync d1.partitions file to VP-Master
[07-09-18 6:19:14 am]  | 32-Dell-790: No need to sync d1p1.img file to VP-Master
[07-09-18 6:28:00 am]  | 32-Dell-790: No need to sync d1p2.img file to VP-Master
[07-09-18 6:28:00 am]  * All files synced for this item.
[07-09-18 6:28:01 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to WCW-Master
[07-09-18 6:28:02 am]  | 32-Dell-790: No need to sync d1.mbr file to WCW-Master
[07-09-18 6:28:02 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to WCW-Master
[07-09-18 6:28:03 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to WCW-Master
[07-09-18 6:28:03 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to WCW-Master
[07-09-18 6:28:04 am]  | 32-Dell-790: No need to sync d1.partitions file to WCW-Master
[07-09-18 6:28:05 am]  | 32-Dell-790: No need to sync d1p1.img file to WCW-Master
[07-09-18 6:34:02 am]  | 32-Dell-790: No need to sync d1p2.img file to WCW-Master
[07-09-18 6:34:02 am]  * All files synced for this item.
[07-09-18 6:34:04 am]  * Found Image to transfer to 8 s
[07-09-18 6:34:04 am]  | Image Name: 64-Dell-790
[07-09-18 6:34:06 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to BCS-Master
[07-09-18 6:34:06 am]  | 64-Dell-790: No need to sync d1.mbr file to BCS-Master
[07-09-18 6:34:07 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to BCS-Master
[07-09-18 6:34:08 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to BCS-Master
[07-09-18 6:34:08 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to BCS-Master
[07-09-18 6:34:09 am]  | 64-Dell-790: No need to sync d1.partitions file to BCS-Master
[07-09-18 6:34:10 am]  | 64-Dell-790: No need to sync d1p1.img file to BCS-Master
[07-09-18 6:41:33 am]  | Files do not match on server: BCS-Master
[07-09-18 6:41:33 am]  | Deleting remote file: /images/64-Dell-790/d1p2.img
[07-09-18 6:41:33 am]  * Starting Sync Actions
[07-09-18 6:41:33 am]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.64-Dell-790.transfer.BCS-Master.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/64-Dell-790" "/images/64-Dell-790"; exit' -u fog,[Protected] 10.210.100.61
[07-09-18 6:41:33 am]  * Started sync for Image 64-Dell-790
[07-09-18 6:41:34 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to CHS-Master
[07-09-18 6:41:34 am]  | 64-Dell-790: No need to sync d1.mbr file to CHS-Master
[07-09-18 6:41:34 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to CHS-Master
[07-09-18 6:41:35 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to CHS-Master
[07-09-18 6:41:35 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to CHS-Master
[07-09-18 6:41:36 am]  | 64-Dell-790: No need to sync d1.partitions file to CHS-Master
[07-09-18 6:41:36 am]  | 64-Dell-790: No need to sync d1p1.img file to CHS-Master
[07-09-18 6:47:24 am]  | 64-Dell-790: No need to sync d1p2.img file to CHS-Master
[07-09-18 6:47:24 am]  * All files synced for this item.
[07-09-18 6:47:25 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to ESC-Master
[07-09-18 6:47:25 am]  | 64-Dell-790: No need to sync d1.mbr file to ESC-Master
[07-09-18 6:47:25 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to ESC-Master
[07-09-18 6:47:25 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to ESC-Master
[07-09-18 6:47:26 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to ESC-Master
[07-09-18 6:47:26 am]  | 64-Dell-790: No need to sync d1.partitions file to ESC-Master
[07-09-18 6:47:26 am]  | 64-Dell-790: No need to sync d1p1.img file to ESC-Master
[07-09-18 6:53:33 am]  | 64-Dell-790: No need to sync d1p2.img file to ESC-Master
[07-09-18 6:53:33 am]  * All files synced for this item.
[07-09-18 6:53:34 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to Jep-Master
[07-09-18 6:53:34 am]  | 64-Dell-790: No need to sync d1.mbr file to Jep-Master
[07-09-18 6:53:34 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to Jep-Master
[07-09-18 6:53:35 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to Jep-Master
[07-09-18 6:53:35 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to Jep-Master
[07-09-18 6:53:36 am]  | 64-Dell-790: No need to sync d1.partitions file to Jep-Master
[07-09-18 6:53:36 am]  | 64-Dell-790: No need to sync d1p1.img file to Jep-Master
[07-09-18 6:59:23 am]  | 64-Dell-790: No need to sync d1p2.img file to Jep-Master
[07-09-18 6:59:23 am]  * All files synced for this item.
[07-09-18 6:59:24 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to VHS-Master
[07-09-18 6:59:24 am]  | 64-Dell-790: No need to sync d1.mbr file to VHS-Master
[07-09-18 6:59:25 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to VHS-Master
[07-09-18 6:59:25 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to VHS-Master
[07-09-18 6:59:26 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to VHS-Master
[07-09-18 6:59:26 am]  | 64-Dell-790: No need to sync d1.partitions file to VHS-Master
[07-09-18 6:59:27 am]  | 64-Dell-790: No need to sync d1p1.img file to VHS-Master
[07-09-18 7:05:52 am]  | 64-Dell-790: No need to sync d1p2.img file to VHS-Master
[07-09-18 7:05:52 am]  * All files synced for this item.
[07-09-18 7:05:53 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to VP-Master
[07-09-18 7:05:54 am]  | 64-Dell-790: No need to sync d1.mbr file to VP-Master
[07-09-18 7:05:55 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to VP-Master
[07-09-18 7:05:55 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to VP-Master
[07-09-18 7:05:56 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to VP-Master
[07-09-18 7:05:57 am]  | 64-Dell-790: No need to sync d1.partitions file to VP-Master
[07-09-18 7:05:58 am]  | 64-Dell-790: No need to sync d1p1.img file to VP-Master
[07-09-18 7:13:12 am]  | Files do not match on server: VP-Master
[07-09-18 7:13:12 am]  | Deleting remote file: /images/64-Dell-790/d1p2.img
[07-09-18 7:13:12 am]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.64-Dell-790.transfer.VP-Master.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/64-Dell-790" "/images/64-Dell-790"; exit' -u fog,[Protected] 10.205.100.61
[07-09-18 7:13:12 am]  * Started sync for Image 64-Dell-790
[07-09-18 7:13:13 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to WCW-Master
[07-09-18 7:13:13 am]  | 64-Dell-790: No need to sync d1.mbr file to WCW-Master
[07-09-18 7:13:14 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to WCW-Master
[07-09-18 7:13:14 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to WCW-Master
[07-09-18 7:13:15 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to WCW-Master
[07-09-18 7:13:15 am]  | 64-Dell-790: No need to sync d1.partitions file to WCW-Master
[07-09-18 7:13:16 am]  | 64-Dell-790: No need to sync d1p1.img file to WCW-Master
[07-09-18 7:19:33 am]  | 64-Dell-790: No need to sync d1p2.img file to WCW-Master
[07-09-18 7:19:33 am]  * All files synced for this item.
[07-09-18 7:19:33 am]  * Attempting to perform Group -> Nodes image replication.
[07-09-18 7:19:33 am]  * Not syncing Image between s
[07-09-18 7:19:33 am]  | Image Name: 32-Dell-790
[07-09-18 7:19:33 am]  | There are no other members to sync to.
[07-09-18 7:19:34 am]  * Not syncing Image between s
[07-09-18 7:19:34 am]  | Image Name: 64-Dell-790
[07-09-18 7:19:34 am]  | There are no other members to sync to.
[07-09-18 9:19:36 am]  * Starting Image Replication.
[07-09-18 9:19:36 am]  * We are group ID: 1. We are group name: default
[07-09-18 9:19:36 am]  * We are node ID: 1. We are node name: DefaultMember
[07-09-18 9:19:36 am]  * Attempting to perform Group -> Group image replication.
[07-09-18 9:19:36 am]  | Replicating postdownloadscripts
[07-09-18 9:19:37 am]  * Not syncing Image between s
[07-09-18 9:19:37 am]  | Image Name:
[07-09-18 9:19:37 am]  | There are no other members to sync to.
[07-09-18 9:19:37 am]  | Replicating postinitscripts
[07-09-18 9:19:37 am]  * Not syncing Image between s
[07-09-18 9:19:37 am]  | Image Name:
[07-09-18 9:19:37 am]  | There are no other members to sync to.
[07-09-18 9:19:40 am]  * Found Image to transfer to 8 s
[07-09-18 9:19:40 am]  | Image Name: 32-Dell-790
[07-09-18 9:19:41 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to BCS-Master
[07-09-18 9:19:42 am]  | 32-Dell-790: No need to sync d1.mbr file to BCS-Master
[07-09-18 9:19:43 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to BCS-Master
[07-09-18 9:19:43 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to BCS-Master
[07-09-18 9:19:44 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to BCS-Master
[07-09-18 9:19:45 am]  | 32-Dell-790: No need to sync d1.partitions file to BCS-Master
[07-09-18 9:19:45 am]  | 32-Dell-790: No need to sync d1p1.img file to BCS-Master
[07-09-18 9:26:50 am]  | 32-Dell-790: No need to sync d1p2.img file to BCS-Master
[07-09-18 9:26:50 am]  * All files synced for this item.
[07-09-18 9:26:50 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to CHS-Master
[07-09-18 9:26:51 am]  | 32-Dell-790: No need to sync d1.mbr file to CHS-Master
[07-09-18 9:26:51 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to CHS-Master
[07-09-18 9:26:51 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to CHS-Master
[07-09-18 9:26:52 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to CHS-Master
[07-09-18 9:26:52 am]  | 32-Dell-790: No need to sync d1.partitions file to CHS-Master
[07-09-18 9:26:53 am]  | 32-Dell-790: No need to sync d1p1.img file to CHS-Master
[07-09-18 9:32:07 am]  | 32-Dell-790: No need to sync d1p2.img file to CHS-Master
[07-09-18 9:32:07 am]  * All files synced for this item.
[07-09-18 9:32:08 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to ESC-Master
[07-09-18 9:32:08 am]  | 32-Dell-790: No need to sync d1.mbr file to ESC-Master
[07-09-18 9:32:08 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to ESC-Master
[07-09-18 9:32:09 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to ESC-Master
[07-09-18 9:32:09 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to ESC-Master
[07-09-18 9:32:09 am]  | 32-Dell-790: No need to sync d1.partitions file to ESC-Master
[07-09-18 9:32:10 am]  | 32-Dell-790: No need to sync d1p1.img file to ESC-Master
[07-09-18 9:36:35 am]  | 32-Dell-790: No need to sync d1p2.img file to ESC-Master
[07-09-18 9:36:35 am]  * All files synced for this item.
[07-09-18 9:36:36 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to Jep-Master
[07-09-18 9:36:36 am]  | 32-Dell-790: No need to sync d1.mbr file to Jep-Master
[07-09-18 9:36:37 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to Jep-Master
[07-09-18 9:36:37 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to Jep-Master
[07-09-18 9:36:37 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to Jep-Master
[07-09-18 9:36:38 am]  | 32-Dell-790: No need to sync d1.partitions file to Jep-Master
[07-09-18 9:36:38 am]  | 32-Dell-790: No need to sync d1p1.img file to Jep-Master
[07-09-18 9:41:56 am]  | 32-Dell-790: No need to sync d1p2.img file to Jep-Master
[07-09-18 9:41:56 am]  * All files synced for this item.
[07-09-18 9:41:58 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VHS-Master
[07-09-18 9:41:59 am]  | 32-Dell-790: No need to sync d1.mbr file to VHS-Master
[07-09-18 9:42:00 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to VHS-Master
[07-09-18 9:42:00 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to VHS-Master
[07-09-18 9:42:01 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to VHS-Master
[07-09-18 9:42:01 am]  | 32-Dell-790: No need to sync d1.partitions file to VHS-Master
[07-09-18 9:42:02 am]  | 32-Dell-790: No need to sync d1p1.img file to VHS-Master
[07-09-18 9:49:02 am]  | 32-Dell-790: No need to sync d1p2.img file to VHS-Master
[07-09-18 9:49:02 am]  * All files synced for this item.
[07-09-18 9:49:03 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VP-Master
[07-09-18 9:49:04 am]  | 32-Dell-790: No need to sync d1.mbr file to VP-Master
[07-09-18 9:49:05 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to VP-Master
[07-09-18 9:49:05 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to VP-Master
[07-09-18 9:49:06 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to VP-Master
[07-09-18 9:49:07 am]  | 32-Dell-790: No need to sync d1.partitions file to VP-Master
[07-09-18 9:49:08 am]  | 32-Dell-790: No need to sync d1p1.img file to VP-Master
[07-09-18 9:56:38 am]  | Files do not match on server: VP-Master
[07-09-18 9:56:38 am]  | Deleting remote file: /images/32-Dell-790/d1p2.img
[07-09-18 9:56:38 am]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.32-Dell-790.transfer.VP-Master.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/32-Dell-790" "/images/32-Dell-790"; exit' -u fog,[Protected] 10.205.100.61
[07-09-18 9:56:38 am]  * Started sync for Image 32-Dell-790
[07-09-18 9:56:38 am]  * Type: 8, File: /var/www/fog/lib/service/fogservice.class.php, Line: 481, Message: Undefined variable: itemType, Host: 10.225.100.61, Username: fog

jflippen

@jgallo @Tom-Elliott
In another thread I was looking into the code a bit and Tom verified that the following code compares the files with one being hashed on one server and the other being hashed on the other.

private static function _filesAreEqual($size_a, $size_b, $file_a, $file_b, $avail) {
        if ($size_a != $size_b) {
            return false;
        }
        if (false === $avail) {
            if ($size_a < 1047685760) {
                $remhash = md5_file($file_b);
                $lochash = md5_file($file_a);
                return ($remhash == $lochash);
            }
            return file_exists($file_b) && file_exists($file_a);
        }
        $hashLoc = self::getHash($file_a);
        $hashRem = $file_b;
        $hashCom = ($hashLoc == $hashRem);
        return $hashCom;

Tom, is it possible that the function is checking for the $hashRem variable before the other server finishes it’s md5hash and therefore comes up with a mismatch? Still doesn’t explain why it won’t delete the file and replace though…

george1421

@jflippen Just as an idea (first let me say I’m not a programmer), if you look about in the code where you can find an example of the replication agent writing to a log file. Clone that and place it in the correct location in the code to write both md5 hash codes into the log. Once the fog server has restarted then it should log that information into the replicator log file. I’ve had to do somethings similar in the past to reverse engineer some of the magic Tom does with his code.

Wayne Workman

@jflippen said in Replication problems 1.5.4 - always copying:

is it possible that the function is checking for the $hashRem variable before the other server finishes it’s md5hash and therefore comes up with a mismatch?

Those calls are blocking-type calls, the code doesn’t move to the next line until they are completed or time out.

jflippen

@wayne-workman Thanks Wayne. Is it possible that the block is timing out then if the file is taking too long on the remote server to create the md5sum and that is why it keeps trying to replace the same file each time there is a replication cycle? Is there a place I can change the time it takes to timeout to try as a troubleshooting technique (or I might go with George’s technique and try and echo out the two variables to a log during that pass to check and see if it is truly grabbing the $hashRem

Tom Elliott

@jflippen PHP is a blocking methodology by its very nature. So it couldn’t continue moving forward if one part is “stuck”. (Unless of course it’s reaching maximum execution timeout, which shouldn’t happen but I don’t know everyone’s environment.)

Tom Elliott

Sorry Read the post and replied, didn’t realize the message had been replied too already.

Sebastian Roth

@ablohowiak @jflippen @Hanz While trying to figure out some replication hickup I found and fixed a couple of issues within the code. All that will be in the next release. Hopefully coming soon. Let me know if you are keen to test those changes beforehand.

jflippen

@Sebastian-Roth Definitely! Though I will probably test on some VM’s first since our actual FOG server is running scripts to back everything up to our nodes instead since we were having issues with both replication services. Will the test code be on the -working channel or a different one? I’ll try to provide feedback when I can. There are other bugs I need to report on as well, but haven’t had time to document.

Sebastian Roth

@jflippen The current changes are on a new branch replication (link) which I will merge into working after a first round of feedback.

Important notice: I had to change some of the hashing code too and therefore nodes being on different versions (1.5.4 or working VS. replication branch) will end up replicating images over and over again. So you need to have all nodes on the replication branch or setup up a separate test environment!!

Please make sure you stop replication first (systemctl stop FOGImageReplicator), then update the storage node and after that update master node.

jflippen

@Sebastian-Roth Okay, so I managed to test out the new branch at home on my VM’s (like I mentioned before, not going to touch our primary FOG server right now sense we have a work around). Here is what I found:

The first time around I tried just doing an in-place upgrade of the working 1.6 branch. This, however seemed to not work properly as it did not change anything after I swapped branches and installed on all three (stopping the service and rebooting on each VM, going from nodes to master).
The 2nd time I did a fresh install of Centos 7 on each VM, followed the wiki guide, and went directly to the replication branch. I then created a fake image by creating a folder and then putting some random files in there (I don’t have PXE boot or a server environment at home). When I checked the log and did some testing, it looks like it’s working! Previously, FOG would not delete mismatched files, meaning the files would never get updated on the node, but would keep trying to run an lftp sync… It would even say files didn’t match even if they had the same md5sum at times. This seems to have been fixed in your new branch! I have included my log below.
During testing I noticed that the replication service won’t remove files that are on the node but not the master (from changing the files I was testing with on the master). Would it be possible to implement an if exist variable that checks an array of the file names in an image folder against the master? This would be handy for keeping the driver injection folder we use in the images folder synced properly too when removing deprecated models.

The Log, as promised

[11-07-18 6:34:23 am] 
==================================
===        ====    =====      ====
===  =========  ==  ===   ==   ===
===  ========  ====  ==  ====  ===
===  ========  ====  ==  =========
===      ====  ====  ==  =========
===  ========  ====  ==  ===   ===
===  ========  ====  ==  ====  ===
===  =========  ==  ===   ==   ===
===  ==========    =====      ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[11-07-18 6:34:23 am] Interface Ready with IP Address: 127.0.0.1
[11-07-18 6:34:23 am] Interface Ready with IP Address: 127.0.1.1
[11-07-18 6:34:23 am] Interface Ready with IP Address: 192.168.1.16
[11-07-18 6:34:23 am] Interface Ready with IP Address: FOG-MASTER
[11-07-18 6:34:23 am] FOGService: ImageReplicator - Waiting for mysql to be available
[11-07-18 6:34:35 am]  * Starting ImageReplicator Service
[11-07-18 6:34:35 am]  * Checking for new items every 600 seconds
[11-07-18 6:34:35 am]  * Starting service loop
[11-07-18 6:34:36 am]  * Starting Image Replication.
[11-07-18 6:34:36 am]  * We are group ID: 1. We are group name: default
[11-07-18 6:34:36 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 6:34:36 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 6:34:36 am]  | Replicating postdownloadscripts
[11-07-18 6:34:38 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:34:38 am]  | File Name: postdownloadscripts
[11-07-18 6:34:38 am]  | NODE-1 server does not appear to be online.
[11-07-18 6:34:38 am]  | NODE-2 server does not appear to be online.
[11-07-18 6:34:38 am]  | Replicating postinitscripts
[11-07-18 6:34:40 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:34:40 am]  | File Name: dev/postinitscripts
[11-07-18 6:34:40 am]  | NODE-1 server does not appear to be online.
[11-07-18 6:34:40 am]  | NODE-2 server does not appear to be online.
[11-07-18 6:34:40 am]  * Not syncing Image between groups
[11-07-18 6:34:40 am]  | Image Name: test
[11-07-18 6:34:40 am]  | There are no other members to sync to.
[11-07-18 6:34:40 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 6:34:42 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:34:42 am]  | Image Name: test
[11-07-18 6:34:42 am]  | NODE-1 server does not appear to be online.
[11-07-18 6:34:42 am]  | NODE-2 server does not appear to be online.
[11-07-18 6:44:42 am]  * Starting Image Replication.
[11-07-18 6:44:42 am]  * We are group ID: 1. We are group name: default
[11-07-18 6:44:42 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 6:44:42 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 6:44:42 am]  | Replicating postdownloadscripts
[11-07-18 6:44:42 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:44:42 am]  | File Name: postdownloadscripts
[11-07-18 6:44:43 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-07-18 6:44:43 am]  * All files synced for this item.
[11-07-18 6:44:43 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-07-18 6:44:43 am]  * All files synced for this item.
[11-07-18 6:44:43 am]  | Replicating postinitscripts
[11-07-18 6:44:44 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:44:44 am]  | File Name: dev/postinitscripts
[11-07-18 6:44:44 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-07-18 6:44:44 am]  * All files synced for this item.
[11-07-18 6:44:45 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-07-18 6:44:45 am]  * All files synced for this item.
[11-07-18 6:44:45 am]  * Not syncing Image between groups
[11-07-18 6:44:45 am]  | Image Name: test
[11-07-18 6:44:45 am]  | There are no other members to sync to.
[11-07-18 6:44:45 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 6:44:46 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:44:46 am]  | Image Name: test
[11-07-18 6:44:46 am]   # test: File does not exist 2014-10-31 11.52.32.jpg (NODE-1)
[11-07-18 6:44:46 am]   # test: File size mismatch - CentOS-7-x86_64-DVD-1804.iso: 4470079488 != 2903506944
[11-07-18 6:44:46 am]   # test: Deleting remote file CentOS-7-x86_64-DVD-1804.iso
[11-07-18 6:44:46 am]   # test: File does not exist Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-07-18 6:44:46 am]   # test: File does not exist FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-07-18 6:44:46 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.test.transfer.NODE-1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.17
[11-07-18 6:44:46 am]  | Started sync for Image test - Resource id #714
[11-07-18 6:44:46 am]   # test: File does not exist 2014-10-31 11.52.32.jpg (NODE-2)
[11-07-18 6:44:46 am]   # test: File size mismatch - CentOS-7-x86_64-DVD-1804.iso: 4470079488 != 2903506944
[11-07-18 6:44:46 am]   # test: Deleting remote file CentOS-7-x86_64-DVD-1804.iso
[11-07-18 6:44:46 am]   # test: File does not exist Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-07-18 6:44:46 am]   # test: File does not exist FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-07-18 6:44:46 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.test.transfer.NODE-2.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.18
[11-07-18 6:44:46 am]  | Started sync for Image test - Resource id #729
[11-07-18 6:48:10 am]  | Sync finished - Resource id #714
[11-07-18 6:48:11 am]  | Sync finished - Resource id #729
[11-07-18 6:54:42 am]  * Starting Image Replication.
[11-07-18 6:54:42 am]  * We are group ID: 1. We are group name: default
[11-07-18 6:54:42 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 6:54:42 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 6:54:42 am]  | Replicating postdownloadscripts
[11-07-18 6:54:43 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:54:43 am]  | File Name: postdownloadscripts
[11-07-18 6:54:43 am]   # postdownloadscripts: File does not exist fog.custominstall (NODE-1)
[11-07-18 6:54:43 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-07-18 6:54:43 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.17
[11-07-18 6:54:43 am]  | Started sync for Image postdownloadscripts - Resource id #781
[11-07-18 6:54:44 am]   # postdownloadscripts: File does not exist fog.custominstall (NODE-2)
[11-07-18 6:54:44 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-07-18 6:54:44 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-2.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.18
[11-07-18 6:54:44 am]  | Started sync for Image postdownloadscripts - Resource id #794
[11-07-18 6:54:44 am]  | Replicating postinitscripts
[11-07-18 6:54:45 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:54:45 am]  | File Name: dev/postinitscripts
[11-07-18 6:54:45 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-07-18 6:54:45 am]  * All files synced for this item.
[11-07-18 6:54:45 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-07-18 6:54:45 am]  * All files synced for this item.
[11-07-18 6:54:46 am]  * Not syncing Image between groups
[11-07-18 6:54:46 am]  | Image Name: test
[11-07-18 6:54:46 am]  | There are no other members to sync to.
[11-07-18 6:54:46 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 6:54:46 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:54:46 am]  | Image Name: test
[11-07-18 6:54:47 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-07-18 6:54:47 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-1)
[11-07-18 6:54:48 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-07-18 6:54:48 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-07-18 6:54:48 am]  * All files synced for this item.
[11-07-18 6:54:49 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-07-18 6:54:49 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-2)
[11-07-18 6:54:49 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-07-18 6:54:50 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-07-18 6:54:50 am]  * All files synced for this item.
[11-07-18 6:54:50 am]  | Sync finished - Resource id #781
[11-07-18 6:54:50 am]  | Sync finished - Resource id #794
[11-07-18 7:04:42 am]  * Starting Image Replication.
[11-07-18 7:04:42 am]  * We are group ID: 1. We are group name: default
[11-07-18 7:04:42 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 7:04:42 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 7:04:42 am]  | Replicating postdownloadscripts
[11-07-18 7:04:42 am]  * Found Image to transfer to 2 nodes
[11-07-18 7:04:42 am]  | File Name: postdownloadscripts
[11-07-18 7:04:42 am]   # postdownloadscripts: File does not exist fog.copydrivers (NODE-1)
[11-07-18 7:04:42 am]   # postdownloadscripts: File size mismatch - fog.custominstall: 1673 != 235
[11-07-18 7:04:42 am]   # postdownloadscripts: Deleting remote file fog.custominstall
[11-07-18 7:04:43 am]   # postdownloadscripts: File size mismatch - fog.postdownload: 271 != 235
[11-07-18 7:04:43 am]   # postdownloadscripts: Deleting remote file fog.postdownload
[11-07-18 7:04:43 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.17
[11-07-18 7:04:43 am]  | Started sync for Image postdownloadscripts - Resource id #975
[11-07-18 7:04:43 am]   # postdownloadscripts: File does not exist fog.copydrivers (NODE-2)
[11-07-18 7:04:43 am]   # postdownloadscripts: File size mismatch - fog.custominstall: 1673 != 235
[11-07-18 7:04:43 am]   # postdownloadscripts: Deleting remote file fog.custominstall
[11-07-18 7:04:43 am]   # postdownloadscripts: File size mismatch - fog.postdownload: 271 != 235
[11-07-18 7:04:43 am]   # postdownloadscripts: Deleting remote file fog.postdownload
[11-07-18 7:04:43 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-2.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.18
[11-07-18 7:04:43 am]  | Started sync for Image postdownloadscripts - Resource id #993
[11-07-18 7:04:43 am]  | Replicating postinitscripts
[11-07-18 7:04:44 am]  * Found Image to transfer to 2 nodes
[11-07-18 7:04:44 am]  | File Name: dev/postinitscripts
[11-07-18 7:04:44 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-07-18 7:04:44 am]  * All files synced for this item.
[11-07-18 7:04:44 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-07-18 7:04:44 am]  * All files synced for this item.
[11-07-18 7:04:45 am]  * Not syncing Image between groups
[11-07-18 7:04:45 am]  | Image Name: test
[11-07-18 7:04:45 am]  | There are no other members to sync to.
[11-07-18 7:04:45 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 7:04:45 am]  * Found Image to transfer to 2 nodes
[11-07-18 7:04:45 am]  | Image Name: test
[11-07-18 7:04:46 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-07-18 7:04:46 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-1)
[11-07-18 7:04:46 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-07-18 7:04:47 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-07-18 7:04:47 am]  * All files synced for this item.
[11-07-18 7:04:47 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-07-18 7:04:47 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-2)
[11-07-18 7:04:48 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-07-18 7:04:48 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-07-18 7:04:48 am]  * All files synced for this item.
[11-07-18 7:04:48 am]  | Sync finished - Resource id #975
[11-07-18 7:04:48 am]  | Sync finished - Resource id #993

Sebastian Roth

@jflippen Thanks heaps for testing and posting feedback! Log’s looking pretty good I reckon.

The first time around I tried just doing an in-place upgrade of the working 1.6 branch. This, however seemed to not work properly as it did not change anything after I swapped branches and installed on all three (stopping the service and rebooting on each VM, going from nodes to master).

This sounds strange. Cannot think of why it shouldn’t properly upgrade from working-1.6 to replication branch. On the other hand I have not tested myself yet. So it’s not impossible to cause an issue. Do you think it’s worth looking into? How would I replicate the issue?

During testing I noticed that the replication service won’t remove files that are on the node but not the master (from changing the files I was testing with on the master). Would it be possible to implement an if exist variable that checks an array of the file names in an image folder against the master? This would be handy for keeping the driver injection folder we use in the images folder synced properly too when removing deprecated models.

Got me! This is one last thing I hope to get implemented before the next release. This has been how replication is working for quite some time as it is not causing trouble in most cases. But you are absolutely right that we should implement that. Will do so!

jflippen

@Sebastian-Roth

This sounds strange. Cannot think of why it shouldn’t properly upgrade from working-1.6 to replication branch. On the other hand I have not tested myself yet. So it’s not impossible to cause an issue. Do you think it’s worth looking into? How would I replicate the issue?

Maybe, but I know it is generally considered best practice to back up the database and start from scratch when it comes to upgrading the FOG server (at least I think I remember reading that somewhere on the wiki or forums). Every time I tried “uninstalling” FOG it is never a clean uninstall. I generally haven’t had issues with doing an in-place update or switching branches, but like I said when i tried to do it for the replication branch going from working 1.6 it kept the new web UI and the log was spitting out the old way, making me believe that either the install skipped files that needed to be replaced or something else failed.

I do have a few other bugs I noticed, but I’ll have to make a new thread for those. I see that there is a bug section in the forum, but wasn’t sure if there was a page with a list of current known bugs or if I just need to search the forum for the bugs I have been experiencing. I don’t want to start a new thread if I don’t need to!

Sebastian Roth

@jflippen Usually just going forward with newer versions by updating the git repo and re-running the installer is working well. Going to working-1.6 and back is a bit different though.

I know there are many more parts of FOG that need fixing (uninstaller is just one of them) but we don’t have as many people helping out with FOG development right now. We hope to see people being interested in FOG to join in and help.

Nevertheless please report bugs as you find them and if they are not answered in the forums yet. But don’t be afraid to open new topics! Do a quick search and if you can’t find anything just post a new topic.

Sebastian Roth

@jflippen Ok, got around to implement that last bit to cleanup files on the storage nodes that might be there from earlier syncs. Simply run git pull to load the fixes and make sure you are still on the right branch via git branch -v. Then re-run the installer and see if that is doing the job properly. Waiting for feedback before I merge this into the working code. Thanks in advance.

jflippen

@Sebastian-Roth Sorry for the delay. I didn’t see your response until yesterday and didn’t get a chance to try until today. So, I did a git pull but the replication branch seems to have vanished! When I checked GitHub it is not present there either. Did you already merge it with the working branch? I am still willing to test it if you have a branch you would like me to use.

jflippen

@Sebastian-Roth Never mind. Answered my own question:
0_1542245282112_a6e2fe70-4d00-4882-b515-6f05f9d91f5e-image.png

I’ll switch branches and do some testing.

jflippen

@Sebastian-Roth aaaand it looks like the updates are now removing files! Good job man! I really appreciate all the hard work you guys are putting into this project. Here is the log from my testing:

[11-14-18 5:42:22 pm] 
==================================
===        ====    =====      ====
===  =========  ==  ===   ==   ===
===  ========  ====  ==  ====  ===
===  ========  ====  ==  =========
===      ====  ====  ==  =========
===  ========  ====  ==  ===   ===
===  ========  ====  ==  ====  ===
===  =========  ==  ===   ==   ===
===  ==========    =====      ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[11-14-18 5:42:22 pm] Interface Ready with IP Address: 127.0.0.1
[11-14-18 5:42:22 pm] Interface Ready with IP Address: 127.0.1.1
[11-14-18 5:42:22 pm] Interface Ready with IP Address: 192.168.1.16
[11-14-18 5:42:22 pm] Interface Ready with IP Address: 192.168.122.1
[11-14-18 5:42:22 pm] Interface Ready with IP Address: FOG-MASTER
[11-14-18 5:42:22 pm]  * Starting ImageReplicator Service
[11-14-18 5:42:22 pm]  * Checking for new items every 600 seconds
[11-14-18 5:42:22 pm]  * Starting service loop
[11-14-18 5:42:22 pm]  * Starting Image Replication.
[11-14-18 5:42:22 pm]  * We are group ID: 1. We are group name: default
[11-14-18 5:42:22 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-14-18 5:42:22 pm]  * Attempting to perform Group -> Group image replication.
[11-14-18 5:42:22 pm]  | Replicating postdownloadscripts
[11-14-18 5:42:23 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:42:23 pm]  | File Name: postdownloadscripts
[11-14-18 5:42:23 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-1)
[11-14-18 5:42:23 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-1)
[11-14-18 5:42:23 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-14-18 5:42:23 pm]  * All files synced for this item.
[11-14-18 5:42:24 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-2)
[11-14-18 5:42:24 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-2)
[11-14-18 5:42:24 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-14-18 5:42:24 pm]  * All files synced for this item.
[11-14-18 5:42:24 pm]  | Replicating postinitscripts
[11-14-18 5:42:24 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:42:24 pm]  | File Name: dev/postinitscripts
[11-14-18 5:42:25 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-14-18 5:42:25 pm]  * All files synced for this item.
[11-14-18 5:42:25 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-14-18 5:42:25 pm]  * All files synced for this item.
[11-14-18 5:42:25 pm]  * Not syncing Image between groups
[11-14-18 5:42:25 pm]  | Image Name: test
[11-14-18 5:42:25 pm]  | There are no other members to sync to.
[11-14-18 5:42:25 pm]  * Attempting to perform Group -> Nodes image replication.
[11-14-18 5:42:26 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:42:26 pm]  | Image Name: test
[11-14-18 5:42:26 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-14-18 5:42:26 pm]   # test: File size mismatch - Fedora-Server-dvd-x86_64-28-1.1.iso: 4470079488 != 2903506944
[11-14-18 5:42:26 pm]   # test: Deleting remote file Fedora-Server-dvd-x86_64-28-1.1.iso
[11-14-18 5:42:26 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-14-18 5:42:26 pm]   # test: File does not exist on master node, deleting CentOS-7-x86_64-DVD-1804.iso on NODE-1
[11-14-18 5:42:26 pm]   # test: File does not exist on master node, deleting rufus-2.12.exe on NODE-1
[11-14-18 5:42:26 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogreplicator.test.transfer.NODE-1.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.17
[11-14-18 5:42:26 pm]  | Started sync for Image test - Resource id #663
[11-14-18 5:42:27 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-14-18 5:42:27 pm]   # test: File size mismatch - Fedora-Server-dvd-x86_64-28-1.1.iso: 4470079488 != 2903506944
[11-14-18 5:42:27 pm]   # test: Deleting remote file Fedora-Server-dvd-x86_64-28-1.1.iso
[11-14-18 5:42:27 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-14-18 5:42:27 pm]   # test: File does not exist on master node, deleting CentOS-7-x86_64-DVD-1804.iso on NODE-2
[11-14-18 5:42:27 pm]   # test: File does not exist on master node, deleting rufus-2.12.exe on NODE-2
[11-14-18 5:42:27 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogreplicator.test.transfer.NODE-2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.18
[11-14-18 5:42:27 pm]  | Started sync for Image test - Resource id #690
[11-14-18 5:45:01 pm]  | Sync finished - Resource id #663
[11-14-18 5:45:01 pm]  | Sync finished - Resource id #690
[11-14-18 5:52:27 pm]  * Starting Image Replication.
[11-14-18 5:52:27 pm]  * We are group ID: 1. We are group name: default
[11-14-18 5:52:27 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-14-18 5:52:27 pm]  * Attempting to perform Group -> Group image replication.
[11-14-18 5:52:27 pm]  | Replicating postdownloadscripts
[11-14-18 5:52:27 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:52:27 pm]  | File Name: postdownloadscripts
[11-14-18 5:52:27 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-1)
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-1)
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-14-18 5:52:28 pm]  * All files synced for this item.
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-2)
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-2)
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-14-18 5:52:28 pm]  * All files synced for this item.
[11-14-18 5:52:28 pm]  | Replicating postinitscripts
[11-14-18 5:52:29 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:52:29 pm]  | File Name: dev/postinitscripts
[11-14-18 5:52:29 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-14-18 5:52:29 pm]  * All files synced for this item.
[11-14-18 5:52:30 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-14-18 5:52:30 pm]  * All files synced for this item.
[11-14-18 5:52:30 pm]  * Not syncing Image between groups
[11-14-18 5:52:30 pm]  | Image Name: test
[11-14-18 5:52:30 pm]  | There are no other members to sync to.
[11-14-18 5:52:30 pm]  * Attempting to perform Group -> Nodes image replication.
[11-14-18 5:52:30 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:52:30 pm]  | Image Name: test
[11-14-18 5:52:31 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-14-18 5:52:31 pm]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-14-18 5:52:31 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-14-18 5:52:31 pm]  * All files synced for this item.
[11-14-18 5:52:32 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-14-18 5:52:32 pm]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-14-18 5:52:32 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-14-18 5:52:32 pm]  * All files synced for this item.
[11-14-18 6:02:27 pm]  * Starting Image Replication.
[11-14-18 6:02:27 pm]  * We are group ID: 1. We are group name: default
[11-14-18 6:02:27 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-14-18 6:02:27 pm]  * Attempting to perform Group -> Group image replication.
[11-14-18 6:02:27 pm]  | Replicating postdownloadscripts
[11-14-18 6:02:27 pm]  * Found Image to transfer to 2 nodes
[11-14-18 6:02:27 pm]  | File Name: postdownloadscripts
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-1)
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-1)
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-14-18 6:02:28 pm]  * All files synced for this item.
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-2)
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-2)
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-14-18 6:02:28 pm]  * All files synced for this item.
[11-14-18 6:02:28 pm]  | Replicating postinitscripts
[11-14-18 6:02:29 pm]  * Found Image to transfer to 2 nodes
[11-14-18 6:02:29 pm]  | File Name: dev/postinitscripts
[11-14-18 6:02:29 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-14-18 6:02:29 pm]  * All files synced for this item.
[11-14-18 6:02:30 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-14-18 6:02:30 pm]  * All files synced for this item.
[11-14-18 6:02:30 pm]  * Not syncing Image between groups
[11-14-18 6:02:30 pm]  | Image Name: test
[11-14-18 6:02:30 pm]  | There are no other members to sync to.
[11-14-18 6:02:30 pm]  * Attempting to perform Group -> Nodes image replication.
[11-14-18 6:02:30 pm]  * Found Image to transfer to 2 nodes
[11-14-18 6:02:30 pm]  | Image Name: test
[11-14-18 6:02:31 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-14-18 6:02:31 pm]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-14-18 6:02:31 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-14-18 6:02:31 pm]  * All files synced for this item.
[11-14-18 6:02:32 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-14-18 6:02:32 pm]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-14-18 6:02:32 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-14-18 6:02:32 pm]  * All files synced for this item.

alt text

Sebastian Roth

@jflippen Yeah, I did move on and merge it back into working as we are preparing for the next release coming fairly soon. Thanks again for testing. Seems fine from the logs.

Replication problems 1.5.4 - always copying

267

11.6k

17.1k

154.6k