FOG Storage Node FTP error in Image Replicator log - files do not match
-
@Tom-Elliott You think the
-in the image name might cause an issue here? Just thinking out loud. -
Highly doubly it. I mean it could happen I suppose? But it’s escaped so I doubt the dash is the issue here.
-
I wanted to tag in and say that I also have this issue. It seems the files transfer fine after I upload an image, but when I check the replicator log I see a never-ending string of files to not match, deleting remote file. I have tried rebooting my master and both storage nodes, checked the md5sum on several files (they matched). I have also tried increasing the time, based off of other threads on the forum. Running 1.5.2 on all three servers. I noticed that even if it says that there is no need to sync, it still goes and says it doesn’t match and deletes the file. /images is set to fog:root -R and permissions to 777 Here is some of my replicator log:
[05-24-18 9:57:43 am] ================================== === ==== ===== ==== === ========= == === == === === ======== ==== == ==== === === ======== ==== == ========= === ==== ==== == ========= === ======== ==== == === === === ======== ==== == ==== === === ========= == === == === === ========== ===== ==== ================================== ===== Free Opensource Ghost ====== ================================== ============ Credits ============= = https://fogproject.org/Credits = ================================== == Released under GPL Version 3 == ================================== [05-24-18 9:57:44 am] Interface Ready with IP Address: 10.59.10.12 [05-24-18 9:57:44 am] Interface Ready with IP Address: 127.0.0.1 [05-24-18 9:57:44 am] Interface Ready with IP Address: 127.0.1.1 [05-24-18 9:57:44 am] Interface Ready with IP Address: do-isdir-01."REDACTED" [05-24-18 9:57:44 am] * Starting ImageReplicator Service [05-24-18 9:57:44 am] * Checking for new items every 10800 seconds [05-24-18 9:57:44 am] * Starting service loop [05-24-18 9:57:45 am] * Starting Image Replication. [05-24-18 9:57:45 am] * We are group ID: 1. We are group name: default [05-24-18 9:57:45 am] * We are node ID: 1. We are node name: DO-FOG-MASTER [05-24-18 9:57:45 am] * Attempting to perform Group -> Group image replication. [05-24-18 9:57:45 am] | Replicating postdownloadscripts [05-24-18 9:57:45 am] * Found Image to transfer to 2 nodes [05-24-18 9:57:45 am] | File Name: postdownloadscripts [05-24-18 9:58:21 am] | postdownloadscripts: No need to sync fog.deletelog file to OC-FOG-NODE [05-24-18 9:58:21 am] | 562 0 /images/postdownloadscripts/fog.deletelog ftp://fog:REDACTED@10.59.181.12/images/postdownloadscripts/fog.deletelog [05-24-18 9:58:21 am] | Files do not match. [05-24-18 9:58:21 am] * Deleting remote file: /images/postdownloadscripts/fog.deletelog [05-24-18 9:58:21 am] | 2258 0 /images/postdownloadscripts/fog.drivers ftp://fog:REDACTED@10.59.181.12/images/postdownloadscripts/fog.drivers [05-24-18 9:58:21 am] | Files do not match. [05-24-18 9:58:21 am] * Deleting remote file: /images/postdownloadscripts/fog.drivers [05-24-18 9:58:21 am] | 2258 0 /images/postdownloadscripts/fog.drivers ftp://fog:REDACTED@10.59.181.12/images/postdownloadscripts/fog.drivers [05-24-18 9:58:21 am] | Files do not match. [05-24-18 9:58:21 am] * Deleting remote file: /images/postdownloadscripts/fog.drivers [05-24-18 9:58:21 am] | 1813 0 /images/postdownloadscripts/fog.postdownload ftp://fog:REDACTED@10.59.181.12/images/postdownloadscripts/fog.postdownload [05-24-18 9:58:21 am] | Files do not match. [05-24-18 9:58:21 am] * Deleting remote file: /images/postdownloadscripts/fog.postdownload [05-24-18 9:58:22 am] | 1813 0 /images/postdownloadscripts/fog.postdownload ftp://fog:REDACTED@10.59.181.12/images/postdownloadscripts/fog.postdownload [05-24-18 9:58:22 am] | Files do not match. [05-24-18 9:58:22 am] * Deleting remote file: /images/postdownloadscripts/fog.postdownload [05-24-18 9:58:22 am] * Starting Sync Actions [05-24-18 9:58:22 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.59.181.12 [05-24-18 9:58:22 am] * Started sync for Image postdownloadscripts [05-24-18 9:58:58 am] | postdownloadscripts: No need to sync fog.deletelog file to PH-FOG-NODE [05-24-18 9:58:58 am] | 562 0 /images/postdownloadscripts/fog.deletelog ftp://fog:REDACTED/images/postdownloadscripts/fog.deletelog [05-24-18 9:58:58 am] | Files do not match. [05-24-18 9:58:58 am] * Deleting remote file: /images/postdownloadscripts/fog.deletelog [05-24-18 9:58:59 am] | 2258 0 /images/postdownloadscripts/fog.drivers ftp://fog:REDACTED/images/postdownloadscripts/fog.drivers [05-24-18 9:58:59 am] | Files do not match. [05-24-18 9:58:59 am] * Deleting remote file: /images/postdownloadscripts/fog.drivers [05-24-18 9:58:59 am] | 2258 0 /images/postdownloadscripts/fog.drivers ftp://fog:REDACTED/images/postdownloadscripts/fog.drivers [05-24-18 9:58:59 am] | Files do not match. [05-24-18 9:58:59 am] * Deleting remote file: /images/postdownloadscripts/fog.drivers [05-24-18 9:58:59 am] | 1813 0 /images/postdownloadscripts/fog.postdownload ftp://fog:REDACTED/images/postdownloadscripts/fog.postdownload [05-24-18 9:58:59 am] | Files do not match. [05-24-18 9:58:59 am] * Deleting remote file: /images/postdownloadscripts/fog.postdownload [05-24-18 9:58:59 am] | 1813 0 /images/postdownloadscripts/fog.postdownload ftp://fog:REDACTED/images/postdownloadscripts/fog.postdownload [05-24-18 9:58:59 am] | Files do not match. [05-24-18 9:58:59 am] * Deleting remote file: /images/postdownloadscripts/fog.postdownload [05-24-18 9:58:59 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.PH-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.60.219.12 [05-24-18 9:58:59 am] * Started sync for Image postdownloadscripts [05-24-18 9:58:59 am] | Replicating postinitscripts [05-24-18 9:58:59 am] * Found Image to transfer to 2 nodes [05-24-18 9:58:59 am] | File Name: dev/postinitscripts [05-24-18 9:59:36 am] | dev/postinitscripts: No need to sync fog.postinit file to OC-FOG-NODE [05-24-18 9:59:36 am] | 249 0 /images/dev/postinitscripts/fog.postinit ftp://fog:REDACTED@10.59.181.12/images/dev/postinitscripts/fog.postinit [05-24-18 9:59:36 am] | Files do not match. [05-24-18 9:59:36 am] * Deleting remote file: /images/dev/postinitscripts/fog.postinit [05-24-18 9:59:36 am] * Starting Sync ActionsIt does this for all the files in my /images folder:
[05-24-18 10:07:57 am] | 15 0 /images/Win7BaseVM/d1.original.fstypes ftp://fog:REDACTED@10.60.219.12/images/Win7BaseVM/d1.original.fstypes [05-24-18 10:07:57 am] | Files do not match. [05-24-18 10:07:57 am] * Deleting remote file: /images/Win7BaseVM/d1.original.fstypes [05-24-18 10:08:15 am] | Win7 Base VM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [05-24-18 10:08:33 am] | Win7 Base VM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [05-24-18 10:08:33 am] | 132 0 /images/Win7BaseVM/d1.partitions ftp://fog:REDACTED@10.60.219.12/images/Win7BaseVM/d1.partitions [05-24-18 10:08:33 am] | Files do not match. [05-24-18 10:08:33 am] * Deleting remote file: /images/Win7BaseVM/d1.partitions [05-24-18 10:08:34 am] | 132 0 /images/Win7BaseVM/d1.partitions ftp://fog:REDACTED@10.60.219.12/images/Win7BaseVM/d1.partitions [05-24-18 10:08:34 am] | Files do not match. [05-24-18 10:08:34 am] * Deleting remote file: /images/Win7BaseVM/d1.partitions [05-24-18 10:09:27 am] | 8925093455 0 /images/Win7BaseVM/d1p1.img ftp://fog:REDACTED@10.60.219.12/images/Win7BaseVM/d1p1.img [05-24-18 10:09:27 am] | Files do not match. [05-24-18 10:09:27 am] * Deleting remote file: /images/Win7BaseVM/d1p1.img [05-24-18 10:10:21 am] | 8925093455 0 /images/Win7BaseVM/d1p1.img ftp://fog:REDACTED@10.60.219.12/images/Win7BaseVM/d1p1.img [05-24-18 10:10:21 am] | Files do not match. [05-24-18 10:10:21 am] * Deleting remote file: /images/Win7BaseVM/d1p1.img [05-24-18 10:10:21 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win7 Base VM.transfer.PH-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win7BaseVM" "/images/Win7BaseVM"; exit' -u fog,[Protected] 10.60.219.12 [05-24-18 10:10:21 am] * Started sync for Image Win7 Base VM -
@jflippen mind updating to working branch and seeing if this helps you out?
-
@tom-elliott Hey Tom, i updated primary FOG server and nodes to working branch version 1.5.2.31 and the issue went away for the most part. It seems one of my nodes still has the same issue, with different files each time:
lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win 10 Base VM.transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win10BaseVM" "/images/Win10BaseVM"; exit' -u fog,[Protected] 10.59.181.12 [05-29-18 6:53:54 am] * Started sync for Image Win 10 Base VM [05-29-18 6:53:54 am] | Replication already running with PID: 1171 [05-29-18 6:53:55 am] * Found Image to transfer to 2 s [05-29-18 6:53:55 am] | Image Name: Win7 Base VM [05-29-18 6:54:04 am] | Win7 Base VM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [05-29-18 6:54:04 am] | Files do not match on server: OC-FOG-NODE [05-29-18 6:54:04 am] | Deleting remote file: /images/Win7BaseVM/d1.mbr [05-29-18 6:54:04 am] | Files do not match on server: OC-FOG-NODE [05-29-18 6:54:04 am] | Deleting remote file: /images/Win7BaseVM/d1.minimum.partitions [05-29-18 6:54:05 am] | Win7 Base VM: No need to sync d1.original.fstypes file to OC-FOG-NODE [05-29-18 6:54:05 am] | Win7 Base VM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [05-29-18 6:54:05 am] | Win7 Base VM: No need to sync d1.partitions file to OC-FOG-NODE [05-29-18 6:56:00 am] | Files do not match on server: OC-FOG-NODE [05-29-18 6:56:00 am] | Deleting remote file: /images/Win7BaseVM/d1p1.img [05-29-18 6:56:00 am] * Starting Sync Actions [05-29-18 6:56:00 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win7 Base VM.transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win7BaseVM" "/images/Win7BaseVM"; exit' -u fog,[Protected] 10.59.181.12 [05-29-18 6:56:00 am] * Started sync for Image Win7 Base VM [05-29-18 6:56:00 am] | Replication already running with PID: 1969Replicator service sleep time is still set for 10800, so I will try re-installing and rebooting on that one.
-
@tom-elliott Hey Tom,
It looks like updating didn’t fix the issue after all. I have all three on 1.5.3 and the issue persists. It seems that the md5sum doesn’t match on the files in the log, and the server keeps trying to sync the same files over and over. I also noticed that whenever the CMD: is sent to transfer files, it will sometimes give an error saying that replication is already running. However, it seems the PID number changes between the error messages.[05-31-18 7:34:02 am] * Starting Image Replication. [05-31-18 7:34:02 am] * We are group ID: 1. We are group name: default [05-31-18 7:34:02 am] * We are node ID: 1. We are node name: DO-FOG-MASTER [05-31-18 7:34:02 am] * Attempting to perform Group -> Group image replication. [05-31-18 7:34:02 am] | Replicating postdownloadscripts [05-31-18 7:34:02 am] * Found Image to transfer to 2 s [05-31-18 7:34:02 am] | File Name: postdownloadscripts [05-31-18 7:34:12 am] | postdownloadscripts: No need to sync fog.deletelog file to OC-FOG-NODE [05-31-18 7:34:12 am] | postdownloadscripts: No need to sync fog.drivers file to OC-FOG-NODE [05-31-18 7:34:13 am] | postdownloadscripts: No need to sync fog.postdownload file to OC-FOG-NODE [05-31-18 7:34:13 am] * All files synced for this item. [05-31-18 7:34:22 am] | postdownloadscripts: No need to sync fog.deletelog file to PH-FOG-NODE [05-31-18 7:34:22 am] | postdownloadscripts: No need to sync fog.drivers file to PH-FOG-NODE [05-31-18 7:34:22 am] | postdownloadscripts: No need to sync fog.postdownload file to PH-FOG-NODE [05-31-18 7:34:22 am] * All files synced for this item. [05-31-18 7:34:22 am] | Replicating postinitscripts [05-31-18 7:34:23 am] * Found Image to transfer to 2 s [05-31-18 7:34:23 am] | File Name: dev/postinitscripts [05-31-18 7:34:32 am] | dev/postinitscripts: No need to sync fog.postinit file to OC-FOG-NODE [05-31-18 7:34:32 am] * All files synced for this item. [05-31-18 7:34:41 am] | dev/postinitscripts: No need to sync fog.postinit file to PH-FOG-NODE [05-31-18 7:34:41 am] * All files synced for this item. [05-31-18 7:34:41 am] * Not syncing Image between s [05-31-18 7:34:41 am] | Image Name: Win 10 Base VM [05-31-18 7:34:41 am] | There are no other members to sync to. [05-31-18 7:34:41 am] * Not syncing Image between s [05-31-18 7:34:41 am] | Image Name: Win7 Base VM [05-31-18 7:34:41 am] | There are no other members to sync to. [05-31-18 7:34:41 am] * Attempting to perform Group -> Nodes image replication. [05-31-18 7:34:42 am] * Found Image to transfer to 2 s [05-31-18 7:34:42 am] | Image Name: Win 10 Base VM [05-31-18 7:34:51 am] | Win 10 Base VM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [05-31-18 7:34:52 am] | Win 10 Base VM: No need to sync d1.mbr file to OC-FOG-NODE [05-31-18 7:34:52 am] | Win 10 Base VM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [05-31-18 7:34:52 am] | Win 10 Base VM: No need to sync d1.original.fstypes file to OC-FOG-NODE [05-31-18 7:34:52 am] | Win 10 Base VM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [05-31-18 7:34:53 am] | Win 10 Base VM: No need to sync d1.partitions file to OC-FOG-NODE [05-31-18 7:34:53 am] | Win 10 Base VM: No need to sync d1p1.img file to OC-FOG-NODE [05-31-18 7:36:38 am] | Files do not match on server: OC-FOG-NODE [05-31-18 7:36:38 am] | Deleting remote file: /images/Win10BaseVM/d1p2.img [05-31-18 7:36:38 am] * Starting Sync Actions [05-31-18 7:36:38 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win 10 Base VM.transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win10BaseVM" "/images/Win10BaseVM"; exit' -u fog,[Protected] 10.59.181.12 [05-31-18 7:36:38 am] * Started sync for Image Win 10 Base VM [05-31-18 7:36:38 am] | Replication already running with PID: 23263 [05-31-18 7:36:38 am] * Found Image to transfer to 2 s [05-31-18 7:36:38 am] | Image Name: Win7 Base VM [05-31-18 7:36:48 am] | Win7 Base VM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [05-31-18 7:36:48 am] | Win7 Base VM: No need to sync d1.mbr file to OC-FOG-NODE [05-31-18 7:36:49 am] | Win7 Base VM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [05-31-18 7:36:49 am] | Win7 Base VM: No need to sync d1.original.fstypes file to OC-FOG-NODE [05-31-18 7:36:49 am] | Win7 Base VM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [05-31-18 7:36:49 am] | Win7 Base VM: No need to sync d1.partitions file to OC-FOG-NODE [05-31-18 7:38:37 am] | Win7 Base VM: No need to sync d1p1.img file to OC-FOG-NODE [05-31-18 7:38:37 am] * All files synced for this item. [05-31-18 7:38:47 am] | Win7 Base VM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE [05-31-18 7:38:47 am] | Files do not match on server: PH-FOG-NODE [05-31-18 7:38:47 am] | Deleting remote file: /images/Win7BaseVM/d1.mbr [05-31-18 7:38:47 am] | Files do not match on server: PH-FOG-NODE [05-31-18 7:38:47 am] | Deleting remote file: /images/Win7BaseVM/d1.minimum.partitions [05-31-18 7:38:47 am] | Win7 Base VM: No need to sync d1.original.fstypes file to PH-FOG-NODE [05-31-18 7:38:48 am] | Win7 Base VM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [05-31-18 7:38:48 am] | Win7 Base VM: No need to sync d1.partitions file to PH-FOG-NODE [05-31-18 7:40:36 am] | Files do not match on server: PH-FOG-NODE [05-31-18 7:40:36 am] | Deleting remote file: /images/Win7BaseVM/d1p1.img [05-31-18 7:40:36 am] * Starting Sync Actions [05-31-18 7:40:36 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win7 Base VM.transfer.PH-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win7BaseVM" "/images/Win7BaseVM"; exit' -u fog,[Protected] 10.60.219.12 [05-31-18 7:40:36 am] * Started sync for Image Win7 Base VM [05-31-18 8:34:02 am] * Starting Image Replication. [05-31-18 8:34:02 am] * We are group ID: 1. We are group name: default [05-31-18 8:34:02 am] * We are node ID: 1. We are node name: DO-FOG-MASTER [05-31-18 8:34:02 am] * Attempting to perform Group -> Group image replication. [05-31-18 8:34:02 am] | Replicating postdownloadscripts [05-31-18 8:34:02 am] * Found Image to transfer to 2 s [05-31-18 8:34:02 am] | File Name: postdownloadscripts [05-31-18 8:34:11 am] | postdownloadscripts: No need to sync fog.deletelog file to OC-FOG-NODE [05-31-18 8:34:12 am] | postdownloadscripts: No need to sync fog.drivers file to OC-FOG-NODE [05-31-18 8:34:12 am] | postdownloadscripts: No need to sync fog.postdownload file to OC-FOG-NODE [05-31-18 8:34:12 am] * All files synced for this item. [05-31-18 8:34:22 am] | postdownloadscripts: No need to sync fog.deletelog file to PH-FOG-NODE [05-31-18 8:34:22 am] | postdownloadscripts: No need to sync fog.drivers file to PH-FOG-NODE [05-31-18 8:34:22 am] | postdownloadscripts: No need to sync fog.postdownload file to PH-FOG-NODE [05-31-18 8:34:22 am] * All files synced for this item. [05-31-18 8:34:22 am] | Replicating postinitscripts [05-31-18 8:34:23 am] * Found Image to transfer to 2 s [05-31-18 8:34:23 am] | File Name: dev/postinitscripts [05-31-18 8:34:31 am] | dev/postinitscripts: No need to sync fog.postinit file to OC-FOG-NODE [05-31-18 8:34:31 am] * All files synced for this item. [05-31-18 8:34:40 am] | dev/postinitscripts: No need to sync fog.postinit file to PH-FOG-NODE [05-31-18 8:34:40 am] * All files synced for this item. [05-31-18 8:34:40 am] * Not syncing Image between s [05-31-18 8:34:40 am] | Image Name: Win 10 Base VM [05-31-18 8:34:40 am] | There are no other members to sync to. [05-31-18 8:34:40 am] * Not syncing Image between s [05-31-18 8:34:40 am] | Image Name: Win7 Base VM [05-31-18 8:34:40 am] | There are no other members to sync to. [05-31-18 8:34:40 am] * Attempting to perform Group -> Nodes image replication. [05-31-18 8:34:41 am] * Found Image to transfer to 2 s [05-31-18 8:34:41 am] | Image Name: Win 10 Base VM [05-31-18 8:34:50 am] | Win 10 Base VM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [05-31-18 8:34:50 am] | Win 10 Base VM: No need to sync d1.mbr file to OC-FOG-NODE [05-31-18 8:34:51 am] | Win 10 Base VM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [05-31-18 8:34:51 am] | Win 10 Base VM: No need to sync d1.original.fstypes file to OC-FOG-NODE [05-31-18 8:34:51 am] | Win 10 Base VM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [05-31-18 8:34:52 am] | Win 10 Base VM: No need to sync d1.partitions file to OC-FOG-NODE [05-31-18 8:34:52 am] | Win 10 Base VM: No need to sync d1p1.img file to OC-FOG-NODE [05-31-18 8:36:36 am] | Files do not match on server: OC-FOG-NODE [05-31-18 8:36:36 am] | Deleting remote file: /images/Win10BaseVM/d1p2.img [05-31-18 8:36:36 am] * Starting Sync Actions [05-31-18 8:36:36 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win 10 Base VM.transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win10BaseVM" "/images/Win10BaseVM"; exit' -u fog,[Protected] 10.59.181.12 [05-31-18 8:36:36 am] * Started sync for Image Win 10 Base VM [05-31-18 8:36:36 am] | Replication already running with PID: 9086 [05-31-18 8:36:37 am] * Found Image to transfer to 2 s [05-31-18 8:36:37 am] | Image Name: Win7 Base VM [05-31-18 8:36:47 am] | Win7 Base VM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [05-31-18 8:36:47 am] | Win7 Base VM: No need to sync d1.mbr file to OC-FOG-NODE [05-31-18 8:36:47 am] | Win7 Base VM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [05-31-18 8:36:48 am] | Win7 Base VM: No need to sync d1.original.fstypes file to OC-FOG-NODE [05-31-18 8:36:48 am] | Win7 Base VM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [05-31-18 8:36:49 am] | Win7 Base VM: No need to sync d1.partitions file to OC-FOG-NODE [05-31-18 8:38:38 am] | Win7 Base VM: No need to sync d1p1.img file to OC-FOG-NODE [05-31-18 8:38:38 am] * All files synced for this item. [05-31-18 8:38:47 am] | Win7 Base VM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE [05-31-18 8:38:48 am] | Files do not match on server: PH-FOG-NODE [05-31-18 8:38:48 am] | Deleting remote file: /images/Win7BaseVM/d1.mbr [05-31-18 8:38:48 am] | Files do not match on server: PH-FOG-NODE [05-31-18 8:38:48 am] | Deleting remote file: /images/Win7BaseVM/d1.minimum.partitions [05-31-18 8:38:48 am] | Win7 Base VM: No need to sync d1.original.fstypes file to PH-FOG-NODE [05-31-18 8:38:48 am] | Win7 Base VM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [05-31-18 8:38:49 am] | Win7 Base VM: No need to sync d1.partitions file to PH-FOG-NODE [05-31-18 8:40:37 am] | Files do not match on server: PH-FOG-NODE [05-31-18 8:40:37 am] | Deleting remote file: /images/Win7BaseVM/d1p1.img [05-31-18 8:40:37 am] * Starting Sync Actions [05-31-18 8:40:37 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win7 Base VM.transfer.PH-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win7BaseVM" "/images/Win7BaseVM"; exit' -u fog,[Protected] 10.60.219.12 [05-31-18 8:40:37 am] * Started sync for Image Win7 Base VMI have also noticed that I have .transfer logs sitting in the log folder for my OC-FOG-NODE, but not my PHS-FOG-NODE, even though the replicator log indicates that it is writing to a transfer log for PHS-FOG-NODE. This is what one of the transfer logs looks like:
2018-05-30 12:34:59 /images/Win10BaseVM/d1.original.fstypes -> ftp://fog@10.59.181.12/%2Fimages/Win10BaseVM/d1.original.fstypes 0-15 2b/s 2018-05-30 12:34:59 /images/Win10BaseVM/d1.partitions -> ftp://fog@10.59.181.12/%2Fimages/Win10BaseVM/d1.partitions 0-190 20b/s 2018-05-30 12:34:59 /images/Win10BaseVM/d1.original.swapuuids -> ftp://fog@10.59.181.12/%2Fimages/Win10BaseVM/d1.original.swapuuids 0-0 2018-05-30 12:34:59 /images/Win10BaseVM/d1.minimum.partitions -> ftp://fog@10.59.181.12/%2Fimages/Win10BaseVM/d1.minimum.partitions 0-190 20b/s 2018-05-30 12:34:59 /images/Win10BaseVM/d1.fixed_size_partitions -> ftp://fog@10.59.181.12/%2Fimages/Win10BaseVM/d1.fixed_size_partitions 0-3 2018-05-30 12:34:59 /images/Win10BaseVM/d1.mbr -> ftp://fog@10.59.181.12/%2Fimages/Win10BaseVM/d1.mbr 0-1048576 108.5K/s 2018-05-30 12:35:00 /images/Win10BaseVM/d1p1.img -> ftp://fog@10.59.181.12/%2Fimages/Win10BaseVM/d1p1.img 0-10220821 994.2K/s 2018-05-30 12:36:27 /images/Win10BaseVM/d1p2.img -> ftp://fog@10.59.181.12/%2Fimages/Win10BaseVM/d1p2.img 0-7248799134 70.58M/s``` -
@jflippen said in FOG Storage Node FTP error in Image Replicator log - files do not match:
Replication already running with PID: 23263
The “replication already running” with the PID changing is to be expected if the prior replication is still running from when it was sent and then being rerun. This is expected behavior. The fact that it appears to be changing which file it needs seems to indicate a problem with the files being checked. This could be that the lftp process isn’t allowed to complete, or a disk is going bad. You even stated this isn’t EVERY file, rather it seems to be random files.
The LFTP Log that you gave is expected. If a file is found that needs to be replicated, the replication tries to send all contents of the folder, though the data inside would not require a lot of time to check and transfer.
As to why the PHS-FOG-NODE isn’t seeing anything, I don’t know. Maybe it can’t actually communicate with that node? (Username/Password/Connectivity Issue?)
-
@tom-elliott Okay, so after some more testing and starting over my OC-FOG-NODE from scratch, I am finding a pattern. It seems the initial transfer of the image goes without any hitches. However, when I update an image, the largest .img file (or any file in which the md5sum changes possibly) seems to never gets updated when the lftp command goes out.
I also noticed that new images have root set for both the owner and group (on the master), even though the /images directory is set to fog:root. permissions are all 777 still though. On the nodes however, the owner and group is fog, but with only execute and read permissions?
I feel that the master node is not able to delete the non-matching .img file from the node before replicating, or isn’t writing over…not sure on the actual process.
Here is Master images folder:
and here is OC node:
This is log before uploading new version of image:
[06-05-18 11:20:00 am] * starting Image Replication. [06-05-18 11:20:00 am] * We are group ID: 1. We are group name: default [06-05-18 11:20:00 am] * We are node ID: 1. We are node name: DO-FOG-MASTER [06-05-18 11:20:00 am] * Attempting to perform Group -> Group image replication. [06-05-18 11:20:00 am] | Replicating postdownloadscripts [06-05-18 11:20:00 am] * Found Image to transfer to 2 s [06-05-18 11:20:00 am] | File Name: postdownloadscripts [06-05-18 11:20:09 am] | postdownloadscripts: No need to sync fog.deletelog file to OC-FOG-NODE [06-05-18 11:20:09 am] | postdownloadscripts: No need to sync fog.drivers file to OC-FOG-NODE [06-05-18 11:20:10 am] | postdownloadscripts: No need to sync fog.postdownload file to OC-FOG-NODE [06-05-18 11:20:10 am] * All files synced for this item. [06-05-18 11:20:19 am] | postdownloadscripts: No need to sync fog.deletelog file to PH-FOG-NODE [06-05-18 11:20:19 am] | postdownloadscripts: No need to sync fog.drivers file to PH-FOG-NODE [06-05-18 11:20:20 am] | postdownloadscripts: No need to sync fog.postdownload file to PH-FOG-NODE [06-05-18 11:20:20 am] * All files synced for this item. [06-05-18 11:20:20 am] | Replicating postinitscripts [06-05-18 11:20:20 am] * Found Image to transfer to 2 s [06-05-18 11:20:20 am] | File Name: dev/postinitscripts [06-05-18 11:20:28 am] | dev/postinitscripts: No need to sync fog.postinit file to OC-FOG-NODE [06-05-18 11:20:28 am] * All files synced for this item. [06-05-18 11:20:37 am] | dev/postinitscripts: No need to sync fog.postinit file to PH-FOG-NODE [06-05-18 11:20:37 am] * All files synced for this item. [06-05-18 11:20:38 am] * Not syncing Image between s [06-05-18 11:20:38 am] | Image Name: Win10BaseVM [06-05-18 11:20:38 am] | There are no other members to sync to. [06-05-18 11:20:38 am] * Not syncing Image between s [06-05-18 11:20:38 am] | Image Name: Win7BaseVM [06-05-18 11:20:38 am] | There are no other members to sync to. [06-05-18 11:20:38 am] * Attempting to perform Group -> Nodes image replication. [06-05-18 11:20:38 am] * Found Image to transfer to 2 s [06-05-18 11:20:38 am] | Image Name: Win10BaseVM [06-05-18 11:20:47 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [06-05-18 11:20:48 am] | Win10BaseVM: No need to sync d1.mbr file to OC-FOG-NODE [06-05-18 11:20:48 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [06-05-18 11:20:48 am] | Win10BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE [06-05-18 11:20:49 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [06-05-18 11:20:49 am] | Win10BaseVM: No need to sync d1.original.uuids file to OC-FOG-NODE [06-05-18 11:20:49 am] | Win10BaseVM: No need to sync d1.partitions file to OC-FOG-NODE [06-05-18 11:20:49 am] | Win10BaseVM: No need to sync d1p1.img file to OC-FOG-NODE [06-05-18 11:20:50 am] | Win10BaseVM: No need to sync d1p2.img file to OC-FOG-NODE [06-05-18 11:20:50 am] | Win10BaseVM: No need to sync d1p3.img file to OC-FOG-NODE [06-05-18 11:22:51 am] | Win10BaseVM: No need to sync d1p4.img file to OC-FOG-NODE [06-05-18 11:22:51 am] * All files synced for this item. [06-05-18 11:23:00 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE [06-05-18 11:23:01 am] | Win10BaseVM: No need to sync d1.mbr file to PH-FOG-NODE [06-05-18 11:23:01 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to PH-FOG-NODE [06-05-18 11:23:01 am] | Win10BaseVM: No need to sync d1.original.fstypes file to PH-FOG-NODE [06-05-18 11:23:02 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [06-05-18 11:23:02 am] | Win10BaseVM: No need to sync d1.original.uuids file to PH-FOG-NODE [06-05-18 11:23:02 am] | Win10BaseVM: No need to sync d1.partitions file to PH-FOG-NODE [06-05-18 11:23:02 am] | Win10BaseVM: No need to sync d1p1.img file to PH-FOG-NODE [06-05-18 11:23:03 am] | Win10BaseVM: No need to sync d1p2.img file to PH-FOG-NODE [06-05-18 11:23:03 am] | Win10BaseVM: No need to sync d1p3.img file to PH-FOG-NODE [06-05-18 11:25:03 am] | Win10BaseVM: No need to sync d1p4.img file to PH-FOG-NODE [06-05-18 11:25:03 am] * All files synced for this item. [06-05-18 11:25:03 am] * Found Image to transfer to 2 s [06-05-18 11:25:03 am] | Image Name: Win7BaseVM [06-05-18 11:25:13 am] | Win7BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [06-05-18 11:25:14 am] | Win7BaseVM: No need to sync d1.mbr file to OC-FOG-NODE [06-05-18 11:25:14 am] | Win7BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [06-05-18 11:25:14 am] | Win7BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE [06-05-18 11:25:14 am] | Win7BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [06-05-18 11:25:15 am] | Win7BaseVM: No need to sync d1.partitions file to OC-FOG-NODE [06-05-18 11:27:12 am] | Win7BaseVM: No need to sync d1p1.img file to OC-FOG-NODE [06-05-18 11:27:12 am] * All files synced for this item. [06-05-18 11:27:21 am] | Win7BaseVM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE [06-05-18 11:27:21 am] | Win7BaseVM: No need to sync d1.mbr file to PH-FOG-NODE [06-05-18 11:27:22 am] | Win7BaseVM: No need to sync d1.minimum.partitions file to PH-FOG-NODE [06-05-18 11:27:22 am] | Win7BaseVM: No need to sync d1.original.fstypes file to PH-FOG-NODE [06-05-18 11:27:22 am] | Win7BaseVM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [06-05-18 11:27:22 am] | Win7BaseVM: No need to sync d1.partitions file to PH-FOG-NODE [06-05-18 11:29:15 am] | Win7BaseVM: No need to sync d1p1.img file to PH-FOG-NODE [06-05-18 11:29:15 am] * All files synced for this item.And this is after uploading my Win7 VM:
[06-05-18 3:20:00 pm] * Starting Image Replication. [06-05-18 3:20:00 pm] * We are group ID: 1. We are group name: default [06-05-18 3:20:00 pm] * We are node ID: 1. We are node name: DO-FOG-MASTER [06-05-18 3:20:00 pm] * Attempting to perform Group -> Group image replication. [06-05-18 3:20:00 pm] | Replicating postdownloadscripts [06-05-18 3:20:00 pm] * Found Image to transfer to 2 s [06-05-18 3:20:00 pm] | File Name: postdownloadscripts [06-05-18 3:20:09 pm] | postdownloadscripts: No need to sync fog.deletelog file to OC-FOG-NODE [06-05-18 3:20:09 pm] | postdownloadscripts: No need to sync fog.drivers file to OC-FOG-NODE [06-05-18 3:20:10 pm] | postdownloadscripts: No need to sync fog.postdownload file to OC-FOG-NODE [06-05-18 3:20:10 pm] * All files synced for this item. [06-05-18 3:20:19 pm] | postdownloadscripts: No need to sync fog.deletelog file to PH-FOG-NODE [06-05-18 3:20:19 pm] | postdownloadscripts: No need to sync fog.drivers file to PH-FOG-NODE [06-05-18 3:20:20 pm] | postdownloadscripts: No need to sync fog.postdownload file to PH-FOG-NODE [06-05-18 3:20:20 pm] * All files synced for this item. [06-05-18 3:20:20 pm] | Replicating postinitscripts [06-05-18 3:20:20 pm] * Found Image to transfer to 2 s [06-05-18 3:20:20 pm] | File Name: dev/postinitscripts [06-05-18 3:20:29 pm] | dev/postinitscripts: No need to sync fog.postinit file to OC-FOG-NODE [06-05-18 3:20:29 pm] * All files synced for this item. [06-05-18 3:20:37 pm] | dev/postinitscripts: No need to sync fog.postinit file to PH-FOG-NODE [06-05-18 3:20:37 pm] * All files synced for this item. [06-05-18 3:20:38 pm] * Not syncing Image between s [06-05-18 3:20:38 pm] | Image Name: Win10BaseVM [06-05-18 3:20:38 pm] | There are no other members to sync to. [06-05-18 3:20:38 pm] * Not syncing Image between s [06-05-18 3:20:38 pm] | Image Name: Win7BaseVM [06-05-18 3:20:38 pm] | There are no other members to sync to. [06-05-18 3:20:38 pm] * Attempting to perform Group -> Nodes image replication. [06-05-18 3:20:38 pm] * Found Image to transfer to 2 s [06-05-18 3:20:38 pm] | Image Name: Win10BaseVM [06-05-18 3:20:48 pm] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [06-05-18 3:20:48 pm] | Win10BaseVM: No need to sync d1.mbr file to OC-FOG-NODE [06-05-18 3:20:48 pm] | Win10BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [06-05-18 3:20:48 pm] | Win10BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE [06-05-18 3:20:49 pm] | Win10BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [06-05-18 3:20:49 pm] | Win10BaseVM: No need to sync d1.original.uuids file to OC-FOG-NODE [06-05-18 3:20:49 pm] | Win10BaseVM: No need to sync d1.partitions file to OC-FOG-NODE [06-05-18 3:20:50 pm] | Win10BaseVM: No need to sync d1p1.img file to OC-FOG-NODE [06-05-18 3:20:50 pm] | Win10BaseVM: No need to sync d1p2.img file to OC-FOG-NODE [06-05-18 3:20:50 pm] | Win10BaseVM: No need to sync d1p3.img file to OC-FOG-NODE [06-05-18 3:22:42 pm] | Win10BaseVM: No need to sync d1p4.img file to OC-FOG-NODE [06-05-18 3:22:42 pm] * All files synced for this item. [06-05-18 3:22:51 pm] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE [06-05-18 3:22:52 pm] | Win10BaseVM: No need to sync d1.mbr file to PH-FOG-NODE [06-05-18 3:22:52 pm] | Win10BaseVM: No need to sync d1.minimum.partitions file to PH-FOG-NODE [06-05-18 3:22:52 pm] | Win10BaseVM: No need to sync d1.original.fstypes file to PH-FOG-NODE [06-05-18 3:22:53 pm] | Win10BaseVM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [06-05-18 3:22:53 pm] | Win10BaseVM: No need to sync d1.original.uuids file to PH-FOG-NODE [06-05-18 3:22:53 pm] | Win10BaseVM: No need to sync d1.partitions file to PH-FOG-NODE [06-05-18 3:22:53 pm] | Win10BaseVM: No need to sync d1p1.img file to PH-FOG-NODE [06-05-18 3:22:54 pm] | Win10BaseVM: No need to sync d1p2.img file to PH-FOG-NODE [06-05-18 3:22:54 pm] | Win10BaseVM: No need to sync d1p3.img file to PH-FOG-NODE [06-05-18 3:24:48 pm] | Win10BaseVM: No need to sync d1p4.img file to PH-FOG-NODE [06-05-18 3:24:48 pm] * All files synced for this item. [06-05-18 3:24:48 pm] * Found Image to transfer to 2 s [06-05-18 3:24:48 pm] | Image Name: Win7BaseVM [06-05-18 3:24:58 pm] | Win7BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [06-05-18 3:24:58 pm] | Win7BaseVM: No need to sync d1.mbr file to OC-FOG-NODE [06-05-18 3:24:59 pm] | Win7BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [06-05-18 3:24:59 pm] | Win7BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE [06-05-18 3:24:59 pm] | Win7BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [06-05-18 3:25:00 pm] | Win7BaseVM: No need to sync d1.partitions file to OC-FOG-NODE [06-05-18 3:26:57 pm] | Files do not match on server: OC-FOG-NODE [06-05-18 3:26:57 pm] | Deleting remote file: /images/Win7BaseVM/d1p1.img [06-05-18 3:26:57 pm] * Starting Sync Actions [06-05-18 3:26:57 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win7BaseVM.transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win7BaseVM" "/images/Win7BaseVM"; exit' -u fog,[Protected] 10.59.181.12 [06-05-18 3:26:57 pm] * Started sync for Image Win7BaseVM [06-05-18 3:26:57 pm] | Replication already running with PID: 25890If I use rsync I can get the files to copy manually, or if I delete the files from the node it seems to be able to send the files out again. It just seems that the replication service won’t update my images. The md5sum on /images/Win7BaseVM.d1p1.img is different between the master and the node, which is triggering the lftp, but it isn’t overriding the node files.
-
@vince-villarreal is your setup still having replication issues as well?
-
@vince-villarreal I have same issue and I have been going through all threads regarding this particular issue. I have exactly the same issue and I feel it’s definitely going through a replication loop. I’ve been monitoring the replication log for past couple days and seems like the replication is stuck on one storage node. Keeps deleting the d1p2.img file and begins synching only to come back to the same file and replicate to same node again.
I should also mention I have reinstalled fog from scratch configured all nodes so initial replication of an uploaded image worked fine. It was indeed after an updated image upload when replication loop began.
