Replication problems 1.5.4 - always copying

JGallo

@ablohowiak Do you think this could be a permissions issue? I think someone mentioned that in a different post but here is my observation from original upload of an image to uploading an updated version of the original image:

permissions of folder on original image - 777 with user:group - fog:fog

permissions of folder after uploading updated image - 755 with user:group - fog:root

I’m currently manually setting permissions back to original settings as if it was a fresh upload then going to upload an updated version and re-check the permissions and see if changes occurred. What i’m hoping for: if changing the permissions of image folders to their original state, then I hope a replication loop doesn’t occur. If it does not, then i will go ahead and upload and continue to observe.

Tom Elliott

@jgallo 755 for fog:root should be fine permissions to allow delete. This is because 7 is the “user” permissions (which is read, write, execute). The first 5 is the group permissions which is read and execute. So the “root” permissions. The second 5 is the “other” which is also read and execute.

I’m not saying you didn’t know this already, just pointing out that the permissions being 777 or 755 should not matter, especially as capture doesn’t have a problem moving the files in place.

Of course any information might help lead to a more suitable solution, but this shouldn’t be a problem at all. It’s worrying as I don’t know where to begin troubleshooting this issue. Especially since the first replication process seems to work perfectly fine.

JGallo

@tom-elliott No worries, I was just throwing some observations out there. My thought process was that since we are dealing with replication and storage groups with various master servers maybe somewhere down the line permissions could be not properly set since if the replication service doesn’t see the file consistency. As I am as lost as any other person with this issue, lol, i’m hoping that maybe some observation input could at least steer you in some direction to troubleshoot.

I just uploaded an image to a location that has a storage group defined and has two storage nodes within that group. So tailing the replication log it eventually shows that files need to replicate. Replication occurs but the log just sits there as if replication occurred to all the nodes. Eventually once replication service restarts from the sleeptime settings in FOG, replication starts fresh and eventually goes back to a loop to same files and same storage node. Hope that helps.

ablohowiak

It doesn’t appear to be a permissions issue. I’m not seeing an entire image as the problem, usually just the largest file, but it’s inconsistent as to which image or storage node will find that the “Files do not match on server”.

[07-06-18 8:25:24 am] * All files synced for this item.
[07-06-18 8:25:25 am] | 000-10-Golden: No need to sync d1.fixed_size_partitions file to FogAllis
[07-06-18 8:25:26 am] | 000-10-Golden: No need to sync d1.mbr file to FogAllis
[07-06-18 8:25:27 am] | 000-10-Golden: No need to sync d1.minimum.partitions file to FogAllis
[07-06-18 8:25:28 am] | 000-10-Golden: No need to sync d1.original.fstypes file to FogAllis
[07-06-18 8:25:28 am] | 000-10-Golden: No need to sync d1.original.swapuuids file to FogAllis
[07-06-18 8:25:29 am] | 000-10-Golden: No need to sync d1.partitions file to FogAllis
[07-06-18 8:25:30 am] | 000-10-Golden: No need to sync d1p1.img file to FogAllis
[07-06-18 8:32:15 am] | Files do not match on server: FogAllis
[07-06-18 8:32:16 am] | Deleting remote file: /images/000-10-Golden/d1p2.img
[07-06-18 8:32:16 am] * Starting Sync Actions
[07-06-18 8:32:16 am] | CMD:
lftp -e 'set xfer:log 1; set xfer:log-file “/opt/fog/log/fogreplicator.000-10-Golden.transfer.FogAllis.log”

drwxrwxrwx 16 fog root 4096 Jun 11 14:12 …
-rwxr-xr-x 1 fog fog 3 Jun 11 10:59 d1.fixed_size_partitions
-rwxr-xr-x 1 fog fog 1048576 Jun 11 10:59 d1.mbr
-rwxr-xr-x 1 fog fog 190 Jun 11 10:59 d1.minimum.partitions
-rwxr-xr-x 1 fog fog 15 Jun 11 10:59 d1.original.fstypes
-rwxr-xr-x 1 fog fog 0 Jun 11 10:59 d1.original.swapuuids
-rwxr-xr-x 1 fog fog 9118461 Jun 11 11:00 d1p1.img
-rwxr-xr-x 1 fog fog 13976945113 Jun 11 12:12 d1p2.img
-rwxr-xr-x 1 fog fog 190 Jun 11 10:59 d1.partitions

But size isn’t the issue, because this is happening with the small files too.

[07-06-18 9:05:29 am] | Files do not match on server: 301Rack
[07-06-18 9:05:29 am] | Deleting remote file: /images/postdownloadscripts/fog.postdownload
[07-06-18 9:05:29 am] | postdownloadscripts: No need to sync fog.postdownload.orig file to 301Rack
[07-06-18 9:05:29 am] * Starting Sync Actions
[07-06-18 9:05:29 am] | CMD:
lftp -e ‘set xfer:log 1; set xfer:log-file “/opt/fog/log/fogreplicator…transfer.301Rack.log”;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude “.srvprivate” “/images/postdownloadscripts” “/images/postdownloadscripts”; exit’ -u fog,[Protected] 10.129.0.17
[07-06-18 9:05:29 am] * Started sync for Image postdownloadscripts
[07-06-18 9:05:29 am] | Replication already running with PID: 8510

The file isn’t deleted, and no transfer log file is ever created. My nodes aren’t all connected at the same speed so I’m going to try throttling the replication to see if it has any impact.

Hanz

@tom-elliott I can verify through multiple attempts that the images are being replicated to the first storage node, but then the replication stops completely. Every other attempt, it seems to want to replicate the biggest (main) partitions as you can see below in the log. d1p3 for UEFI or d1p2 for MBR images.

[07-07-18 12:19:26 pm] | There are no other members to sync to.
[07-07-18 12:19:26 pm] | Image Name: SysprepWin10EDU
[07-07-18 12:19:26 pm] * Not syncing Image between s
[07-07-18 12:19:26 pm] | There are no other members to sync to.
[07-07-18 12:19:26 pm] | Image Name: Sysprep-Win10EDU-X64
[07-07-18 12:19:26 pm] * Not syncing Image between s
[07-07-18 12:19:25 pm] | There are no other members to sync to.
[07-07-18 12:19:25 pm] | Image Name: Sysprep-UEFI-x64
[07-07-18 12:19:25 pm] * Not syncing Image between s
[07-07-18 12:19:25 pm] | There are no other members to sync to.
[07-07-18 12:19:25 pm] | Image Name: Sysprep-All (Win 7)
[07-07-18 12:19:25 pm] * Not syncing Image between s
[07-07-18 12:19:24 pm] * Attempting to perform Group -> Nodes image replication.
[07-07-18 12:19:24 pm] | Replication already running with PID: 40445
[07-07-18 12:19:24 pm] | Replication already running with PID: 40445
[07-07-18 12:19:24 pm] | Replication already running with PID: 40445
[07-07-18 12:19:24 pm] | Replication already running with PID: 40445
[07-07-18 12:19:24 pm] | Replication already running with PID: 40445
[07-07-18 12:19:24 pm] | Replication already running with PID: 40445
[07-07-18 12:19:24 pm] | Replication already running with PID: 40445
[07-07-18 12:19:24 pm] * Started sync for Image SysprepWin10EDU
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.SysprepWin10EDU.transfer.Ashford.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepWin10EDU" "/images/SysprepWin10EDU"; exit' -u fog,[Protected] 10.72.128.55
[07-07-18 12:19:24 pm] | CMD:
[07-07-18 12:19:24 pm] * Starting Sync Actions
[07-07-18 12:19:24 pm] | Deleting remote file: /images/SysprepWin10EDU/d1p1.img
[07-07-18 12:19:24 pm] | Files do not match on server: Ashford
[07-07-18 12:17:57 pm] | SysprepWin10EDU: No need to sync d1.partitions file to Ashford
[07-07-18 12:17:57 pm] | SysprepWin10EDU: No need to sync d1.original.swapuuids file to Ashford
[07-07-18 12:17:56 pm] | SysprepWin10EDU: No need to sync d1.original.fstypes file to Ashford
[07-07-18 12:17:56 pm] | SysprepWin10EDU: No need to sync d1.minimum.partitions file to Ashford
[07-07-18 12:17:55 pm] | SysprepWin10EDU: No need to sync d1.mbr file to Ashford
[07-07-18 12:17:54 pm] | SysprepWin10EDU: No need to sync d1.fixed_size_partitions file to Ashford
[07-07-18 12:17:53 pm] | Image Name: SysprepWin10EDU
[07-07-18 12:17:53 pm] * Found Image to transfer to 9 s
[07-07-18 12:17:50 pm] | Replication already running with PID: 28731
[07-07-18 12:17:50 pm] | Replication already running with PID: 28731
[07-07-18 12:17:50 pm] | Replication already running with PID: 28731
[07-07-18 12:17:50 pm] | Replication already running with PID: 28731
[07-07-18 12:17:50 pm] | Replication already running with PID: 28731
[07-07-18 12:17:50 pm] | Replication already running with PID: 28731
[07-07-18 12:17:50 pm] | Replication already running with PID: 28731
[07-07-18 12:17:50 pm] * Started sync for Image Sysprep-Win10EDU-X64
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-Win10EDU-X64.transfer.Ashford.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepImagex64" "/images/SysprepImagex64"; exit' -u fog,[Protected] 10.72.128.55
[07-07-18 12:17:50 pm] | CMD:
[07-07-18 12:17:50 pm] * Starting Sync Actions
[07-07-18 12:17:50 pm] | Deleting remote file: /images/SysprepImagex64/d1p2.img
[07-07-18 12:17:50 pm] | Files do not match on server: Ashford
[07-07-18 12:16:03 pm] | Sysprep-Win10EDU-X64: No need to sync d1p1.img file to Ashford
[07-07-18 12:16:02 pm] | Sysprep-Win10EDU-X64: No need to sync d1.partitions file to Ashford
[07-07-18 12:16:01 pm] | Sysprep-Win10EDU-X64: No need to sync d1.original.swapuuids file to Ashford
[07-07-18 12:16:01 pm] | Sysprep-Win10EDU-X64: No need to sync d1.original.fstypes file to Ashford
[07-07-18 12:16:00 pm] | Sysprep-Win10EDU-X64: No need to sync d1.minimum.partitions file to Ashford
[07-07-18 12:16:00 pm] | Sysprep-Win10EDU-X64: No need to sync d1.mbr file to Ashford
[07-07-18 12:15:59 pm] | Sysprep-Win10EDU-X64: No need to sync d1.fixed_size_partitions file to Ashford
[07-07-18 12:15:58 pm] | Image Name: Sysprep-Win10EDU-X64
[07-07-18 12:15:58 pm] * Found Image to transfer to 9 s
[07-07-18 12:15:55 pm] | Replication already running with PID: 14564
[07-07-18 12:15:55 pm] | Replication already running with PID: 14564
[07-07-18 12:15:55 pm] | Replication already running with PID: 14564
[07-07-18 12:15:55 pm] | Replication already running with PID: 14564
[07-07-18 12:15:55 pm] | Replication already running with PID: 14564
[07-07-18 12:15:55 pm] | Replication already running with PID: 14564
[07-07-18 12:15:55 pm] | Replication already running with PID: 14564
[07-07-18 12:15:55 pm] * Started sync for Image Sysprep-UEFI-x64
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-UEFI-x64.transfer.Ashford.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Sysprep-UEFI-x64" "/images/Sysprep-UEFI-x64"; exit' -u fog,[Protected] 10.72.128.55
[07-07-18 12:15:55 pm] | CMD:
[07-07-18 12:15:55 pm] * Starting Sync Actions
[07-07-18 12:15:55 pm] | Sysprep-UEFI-x64: No need to sync d1p4.img file to Ashford
[07-07-18 12:15:54 pm] | Deleting remote file: /images/Sysprep-UEFI-x64/d1p3.img
[07-07-18 12:15:54 pm] | Files do not match on server: Ashford
[07-07-18 12:13:49 pm] | Sysprep-UEFI-x64: No need to sync d1p2.img file to Ashford
[07-07-18 12:13:48 pm] | Sysprep-UEFI-x64: No need to sync d1p1.img file to Ashford
[07-07-18 12:13:47 pm] | Sysprep-UEFI-x64: No need to sync d1.partitions file to Ashford
[07-07-18 12:13:47 pm] | Sysprep-UEFI-x64: No need to sync d1.original.uuids file to Ashford
[07-07-18 12:13:46 pm] | Sysprep-UEFI-x64: No need to sync d1.original.swapuuids file to Ashford
[07-07-18 12:13:46 pm] | Sysprep-UEFI-x64: No need to sync d1.original.fstypes file to Ashford
[07-07-18 12:13:45 pm] | Sysprep-UEFI-x64: No need to sync d1.minimum.partitions file to Ashford
[07-07-18 12:13:45 pm] | Sysprep-UEFI-x64: No need to sync d1.mbr file to Ashford
[07-07-18 12:13:44 pm] | Sysprep-UEFI-x64: No need to sync d1.fixed_size_partitions file to Ashford
[07-07-18 12:13:42 pm] | Image Name: Sysprep-UEFI-x64
[07-07-18 12:13:42 pm] * Found Image to transfer to 9 s
[07-07-18 12:13:39 pm] | There are no other members to sync to.
[07-07-18 12:13:39 pm] | Image Name: Sysprep-All (Win 7)
[07-07-18 12:13:39 pm] * Not syncing Image between s
[07-07-18 12:13:39 pm] | There are no other members to sync to.
[07-07-18 12:13:39 pm] | Image Name: 
[07-07-18 12:13:39 pm] * Not syncing Image between s
[07-07-18 12:13:38 pm] | Replicating postinitscripts
[07-07-18 12:13:38 pm] | There are no other members to sync to.
[07-07-18 12:13:38 pm] | Image Name: 
[07-07-18 12:13:38 pm] * Not syncing Image between s
[07-07-18 12:13:38 pm] | Replicating postdownloadscripts
[07-07-18 12:13:38 pm] * Attempting to perform Group -> Group image replication.
[07-07-18 12:13:38 pm] * We are node ID: 1. We are node name: 1Fog-Server
[07-07-18 12:13:38 pm] * We are group ID: 1. We are group name: 1Office
[07-07-18 12:13:38 pm] * Starting Image Replication.

FOG 1.5.4.8

Furthermore…after disabling the Ashford node it does in fact go to the second “Ramage” node, but stops there and doesn’t try to replicate to the other nodes whatsoever…(I have 9 nodes alltogether, including the main server)

[07-07-18 1:11:30 pm] | There are no other members to sync to.
[07-07-18 1:11:30 pm] | Image Name: SysprepWin10EDU
[07-07-18 1:11:30 pm] * Not syncing Image between s
[07-07-18 1:11:30 pm] | There are no other members to sync to.
[07-07-18 1:11:30 pm] | Image Name: Sysprep-Win10EDU-X64
[07-07-18 1:11:29 pm] * Not syncing Image between s
[07-07-18 1:11:29 pm] | There are no other members to sync to.
[07-07-18 1:11:29 pm] | Image Name: Sysprep-UEFI-x64
[07-07-18 1:11:29 pm] * Not syncing Image between s
[07-07-18 1:11:29 pm] | There are no other members to sync to.
[07-07-18 1:11:28 pm] | Image Name: Sysprep-All (Win 7)
[07-07-18 1:11:28 pm] * Not syncing Image between s
[07-07-18 1:11:28 pm] * Attempting to perform Group -> Nodes image replication.
[07-07-18 1:11:28 pm] | Replication already running with PID: 55853
[07-07-18 1:11:28 pm] | Replication already running with PID: 55853
[07-07-18 1:11:28 pm] | Replication already running with PID: 55853
[07-07-18 1:11:28 pm] | Replication already running with PID: 55853
[07-07-18 1:11:28 pm] | Replication already running with PID: 55853
[07-07-18 1:11:28 pm] | Replication already running with PID: 55853
[07-07-18 1:11:28 pm] * Started sync for Image SysprepWin10EDU
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.SysprepWin10EDU.transfer.Ramage.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepWin10EDU" "/images/SysprepWin10EDU"; exit' -u fog,[Protected] 10.72.216.50
[07-07-18 1:11:28 pm] | CMD:
[07-07-18 1:11:28 pm] * Starting Sync Actions
[07-07-18 1:11:28 pm] | d1p1.img File does not exist. Ramage
[07-07-18 1:11:28 pm] | Files do not match on server: Ramage
[07-07-18 1:11:20 pm] | d1.partitions File does not exist. Ramage
[07-07-18 1:11:20 pm] | Files do not match on server: Ramage
[07-07-18 1:11:13 pm] | d1.original.swapuuids File does not exist. Ramage
[07-07-18 1:11:13 pm] | Files do not match on server: Ramage
[07-07-18 1:11:09 pm] | d1.original.fstypes File does not exist. Ramage
[07-07-18 1:11:09 pm] | Files do not match on server: Ramage
[07-07-18 1:11:05 pm] | d1.minimum.partitions File does not exist. Ramage
[07-07-18 1:11:05 pm] | Files do not match on server: Ramage
[07-07-18 1:10:59 pm] | d1.mbr File does not exist. Ramage
[07-07-18 1:10:59 pm] | Files do not match on server: Ramage
[07-07-18 1:10:51 pm] | d1.fixed_size_partitions File does not exist. Ramage
[07-07-18 1:10:51 pm] | Files do not match on server: Ramage
[07-07-18 1:10:40 pm] | Image Name: SysprepWin10EDU
[07-07-18 1:10:40 pm] * Found Image to transfer to 8 s
[07-07-18 1:10:33 pm] | Replication already running with PID: 51510
[07-07-18 1:10:33 pm] | Replication already running with PID: 51510
[07-07-18 1:10:33 pm] | Replication already running with PID: 51510
[07-07-18 1:10:33 pm] | Replication already running with PID: 51510
[07-07-18 1:10:33 pm] | Replication already running with PID: 51510
[07-07-18 1:10:33 pm] | Replication already running with PID: 51510
[07-07-18 1:10:33 pm] * Started sync for Image Sysprep-Win10EDU-X64
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-Win10EDU-X64.transfer.Ramage.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepImagex64" "/images/SysprepImagex64"; exit' -u fog,[Protected] 10.72.216.50
[07-07-18 1:10:33 pm] | CMD:
[07-07-18 1:10:33 pm] * Starting Sync Actions
[07-07-18 1:10:33 pm] | d1p2.img File does not exist. Ramage
[07-07-18 1:10:33 pm] | Files do not match on server: Ramage
[07-07-18 1:10:28 pm] | d1p1.img File does not exist. Ramage
[07-07-18 1:10:28 pm] | Files do not match on server: Ramage
[07-07-18 1:10:24 pm] | d1.partitions File does not exist. Ramage
[07-07-18 1:10:24 pm] | Files do not match on server: Ramage
[07-07-18 1:10:20 pm] | d1.original.swapuuids File does not exist. Ramage
[07-07-18 1:10:20 pm] | Files do not match on server: Ramage
[07-07-18 1:10:14 pm] | d1.original.fstypes File does not exist. Ramage
[07-07-18 1:10:14 pm] | Files do not match on server: Ramage
[07-07-18 1:10:11 pm] | d1.minimum.partitions File does not exist. Ramage
[07-07-18 1:10:11 pm] | Files do not match on server: Ramage
[07-07-18 1:10:09 pm] | d1.mbr File does not exist. Ramage
[07-07-18 1:10:09 pm] | Files do not match on server: Ramage
[07-07-18 1:10:07 pm] | d1.fixed_size_partitions File does not exist. Ramage
[07-07-18 1:10:07 pm] | Files do not match on server: Ramage
[07-07-18 1:10:01 pm] | Image Name: Sysprep-Win10EDU-X64
[07-07-18 1:10:01 pm] * Found Image to transfer to 8 s
[07-07-18 1:09:57 pm] | Replication already running with PID: 48074
[07-07-18 1:09:57 pm] | Replication already running with PID: 48074
[07-07-18 1:09:57 pm] | Replication already running with PID: 48074
[07-07-18 1:09:57 pm] | Replication already running with PID: 48074
[07-07-18 1:09:57 pm] | Replication already running with PID: 48074
[07-07-18 1:09:57 pm] | Replication already running with PID: 48074
[07-07-18 1:09:57 pm] * Started sync for Image Sysprep-UEFI-x64
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-UEFI-x64.transfer.Ramage.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Sysprep-UEFI-x64" "/images/Sysprep-UEFI-x64"; exit' -u fog,[Protected] 10.72.216.50
[07-07-18 1:09:57 pm] | CMD:
[07-07-18 1:09:57 pm] * Starting Sync Actions
[07-07-18 1:09:57 pm] | d1p4.img File does not exist. Ramage
[07-07-18 1:09:57 pm] | Files do not match on server: Ramage
[07-07-18 1:09:56 pm] | d1p3.img File does not exist. Ramage
[07-07-18 1:09:56 pm] | Files do not match on server: Ramage
[07-07-18 1:09:56 pm] | d1p2.img File does not exist. Ramage
[07-07-18 1:09:56 pm] | Files do not match on server: Ramage
[07-07-18 1:09:55 pm] | d1p1.img File does not exist. Ramage
[07-07-18 1:09:55 pm] | Files do not match on server: Ramage
[07-07-18 1:09:55 pm] | d1.partitions File does not exist. Ramage
[07-07-18 1:09:55 pm] | Files do not match on server: Ramage
[07-07-18 1:09:54 pm] | d1.original.uuids File does not exist. Ramage
[07-07-18 1:09:54 pm] | Files do not match on server: Ramage
[07-07-18 1:09:54 pm] | d1.original.swapuuids File does not exist. Ramage
[07-07-18 1:09:54 pm] | Files do not match on server: Ramage
[07-07-18 1:09:53 pm] | d1.original.fstypes File does not exist. Ramage
[07-07-18 1:09:53 pm] | Files do not match on server: Ramage
[07-07-18 1:09:53 pm] | d1.minimum.partitions File does not exist. Ramage
[07-07-18 1:09:53 pm] | Files do not match on server: Ramage
[07-07-18 1:09:52 pm] | d1.mbr File does not exist. Ramage
[07-07-18 1:09:52 pm] | Files do not match on server: Ramage
[07-07-18 1:09:52 pm] | d1.fixed_size_partitions File does not exist. Ramage
[07-07-18 1:09:52 pm] | Files do not match on server: Ramage
[07-07-18 1:09:50 pm] | Image Name: Sysprep-UEFI-x64
[07-07-18 1:09:50 pm] * Found Image to transfer to 8 s
[07-07-18 1:09:48 pm] | There are no other members to sync to.
[07-07-18 1:09:48 pm] | Image Name: Sysprep-All (Win 7)
[07-07-18 1:09:48 pm] * Not syncing Image between s
[07-07-18 1:09:47 pm] | There are no other members to sync to.
[07-07-18 1:09:47 pm] | Image Name: 
[07-07-18 1:09:47 pm] * Not syncing Image between s
[07-07-18 1:09:47 pm] | Replicating postinitscripts
[07-07-18 1:09:47 pm] | There are no other members to sync to.
[07-07-18 1:09:47 pm] | Image Name: 
[07-07-18 1:09:47 pm] * Not syncing Image between s
[07-07-18 1:09:46 pm] | Replicating postdownloadscripts
[07-07-18 1:09:46 pm] * Attempting to perform Group -> Group image replication.
[07-07-18 1:09:46 pm] * We are node ID: 1. We are node name: 1Fog-Server
[07-07-18 1:09:46 pm] * We are group ID: 1. We are group name: 1Office
[07-07-18 1:09:46 pm] * Starting Image Replication.
[07-07-18 1:09:43 pm] * Starting service loop
[07-07-18 1:09:43 pm] * Checking for new items every 4800 seconds
[07-07-18 1:09:43 pm] * Starting ImageReplicator Service
[07-07-18 1:09:43 pm] Interface Ready with IP Address: fog-server.boone.edu
[07-07-18 1:09:43 pm] Interface Ready with IP Address: 127.0.1.1
[07-07-18 1:09:43 pm] Interface Ready with IP Address: 127.0.0.1
[07-07-18 1:09:43 pm] Interface Ready with IP Address: 10.72.3.50


==================================
== Released under GPL Version 3 ==
==================================
= https://fogproject.org/Credits =
============ Credits =============
==================================
===== Free Opensource Ghost ======
==================================
=== ========== ===== ====
=== ========= == === == ===
=== ======== ==== == ==== ===
=== ======== ==== == === ===
=== ==== ==== == =========
=== ======== ==== == =========
=== ======== ==== == ==== ===
=== ========= == === == ===
=== ==== ===== ====
==================================
[07-07-18 1:09:43 pm]

After disabling each node in order after replication, I can get images replicated to all nodes.

Tom Elliott

@hanz et al.

Mind updating again to latest working, I didn’t push a version change yet, I just pushed a quick fix to what I hope will help out.

Essentially, I’m setting the variable for checking running processes. However, the variable was accidentally unset within its running scope, so I re-added that variable so it should be used. Maybe this is why the weirdness was happening? It was always checking the running process of the first item in the list, so while it did the checks on the other nodes, only the first node was being checked for a running process (hence why disabling that first node would allow the next, and so forth, to start replicating properly.)

It’s a long shot, and I don’t have a means to test myself. Please just run a git pull and see if that helps out at all.

Hanz

@tom-elliott You are the man sir…The replication does indeed appear to be working as expected…I removed files from one node to confirm complete transfer. The only question I have concerns the odd output in Image Replicator log showing the following:

[07-09-18 9:22:55 am] * Started sync for Image Sysprep-Win10EDU-X64
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-Win10EDU-X64.transfer.VanHigh.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepImagex64" "/images/SysprepImagex64"; exit' -u fog,[Protected] 10.73.176.50
[07-09-18 9:22:55 am] | CMD:
[07-09-18 9:22:55 am] | Deleting remote file: /images/SysprepImagex64/d1p2.img
[07-09-18 9:22:55 am] | Files do not match on server: VanHigh
[07-09-18 9:21:02 am] | Sysprep-Win10EDU-X64: No need to sync d1p1.img file to VanHigh
[07-09-18 9:21:01 am] | Sysprep-Win10EDU-X64: No need to sync d1.partitions file to VanHigh
[07-09-18 9:21:01 am] | Sysprep-Win10EDU-X64: No need to sync d1.original.swapuuids file to VanHigh
[07-09-18 9:21:01 am] | Sysprep-Win10EDU-X64: No need to sync d1.original.fstypes file to VanHigh
[07-09-18 9:21:00 am] | Sysprep-Win10EDU-X64: No need to sync d1.minimum.partitions file to VanHigh
[07-09-18 9:21:00 am] | Sysprep-Win10EDU-X64: No need to sync d1.mbr file to VanHigh
[07-09-18 9:20:59 am] | Sysprep-Win10EDU-X64: No need to sync d1.fixed_size_partitions file to VanHigh
[07-09-18 9:20:58 am] * Started sync for Image Sysprep-Win10EDU-X64
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-Win10EDU-X64.transfer.VanElem.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepImagex64" "/images/SysprepImagex64"; exit' -u fog,[Protected] 10.72.200.50
[07-09-18 9:20:58 am] | CMD:
[07-09-18 9:20:58 am] | Deleting remote file: /images/SysprepImagex64/d1p2.img
[07-09-18 9:20:58 am] | Files do not match on server: VanElem
[07-09-18 9:18:57 am] | Sysprep-Win10EDU-X64: No need to sync d1p1.img file to VanElem
[07-09-18 9:18:55 am] | Sysprep-Win10EDU-X64: No need to sync d1.partitions file to VanElem
[07-09-18 9:18:54 am] | Sysprep-Win10EDU-X64: No need to sync d1.original.swapuuids file to VanElem
[07-09-18 9:18:53 am] | Sysprep-Win10EDU-X64: No need to sync d1.original.fstypes file to VanElem
[07-09-18 9:18:53 am] | Sysprep-Win10EDU-X64: No need to sync d1.minimum.partitions file to VanElem
[07-09-18 9:18:52 am] | Sysprep-Win10EDU-X64: No need to sync d1.mbr file to VanElem
[07-09-18 9:18:51 am] | Sysprep-Win10EDU-X64: No need to sync d1.fixed_size_partitions file to VanElem

The files are there already and not actually being deleted and resent as the log indicates…I only attached the logs from 2 nodes, but they all behave the same and have the same output when image is already present. This does seem to only affect the largest partition.

jflippen

@hanz @Tom-Elliott I updated this morning and can confirm that my server will now check other nodes after the first one in line has files to replace. However, I believe it is still having the issue of the files saying they are different even though the md5sum is the same (and not deleting said file when it says it will and transfer to log)
0_1531146919906_e5fb0a00-8b12-44c4-91ae-7e8efc4b206d-image.png

[07-09-18 7:04:19 am] 
==================================
=== ==== ===== ====
=== ========= == === == ===
=== ======== ==== == ==== ===
=== ======== ==== == =========
=== ==== ==== == =========
=== ======== ==== == === ===
=== ======== ==== == ==== ===
=== ========= == === == ===
=== ========== ===== ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[07-09-18 7:04:19 am] Interface Ready with IP Address: 10.59.10.12
[07-09-18 7:04:19 am] Interface Ready with IP Address: 127.0.0.1
[07-09-18 7:04:19 am] Interface Ready with IP Address: 127.0.1.1
[07-09-18 7:04:19 am] Interface Ready with IP Address: 192.168.122.1
[07-09-18 7:04:19 am] Interface Ready with IP Address: do-isdir-01.tsd.tsdnet.turlock.k12.ca.us
[07-09-18 7:04:19 am] Interface Ready with IP Address: fog-master-tsd-tsdnet-turlock-k12-ca-us
[07-09-18 7:04:19 am] * Starting ImageReplicator Service
[07-09-18 7:04:19 am] * Checking for new items every 900 seconds
[07-09-18 7:04:19 am] * Starting service loop
[07-09-18 7:04:19 am] * Starting Image Replication.
[07-09-18 7:04:19 am] * We are group ID: 1. We are group name: default
[07-09-18 7:04:19 am] * We are node ID: 1. We are node name: DO-FOG-MASTER
[07-09-18 7:04:19 am] * Attempting to perform Group -> Group image replication.
[07-09-18 7:04:19 am] | Replicating postdownloadscripts
[07-09-18 7:04:20 am] * Found Image to transfer to 2 s
[07-09-18 7:04:20 am] | File Name: postdownloadscripts
[07-09-18 7:04:29 am] | postdownloadscripts: No need to sync fog.deletelog file to OC-FOG-NODE
[07-09-18 7:04:29 am] | postdownloadscripts: No need to sync fog.drivers file to OC-FOG-NODE
[07-09-18 7:04:29 am] | postdownloadscripts: No need to sync fog.postdownload file to OC-FOG-NODE
[07-09-18 7:04:29 am] * All files synced for this item.
[07-09-18 7:04:48 am] | postdownloadscripts: No need to sync fog.deletelog file to PH-FOG-NODE
[07-09-18 7:04:48 am] | Files do not match on server: PH-FOG-NODE
[07-09-18 7:04:48 am] | Deleting remote file: /images/postdownloadscripts/fog.drivers
[07-09-18 7:04:48 am] | Files do not match on server: PH-FOG-NODE
[07-09-18 7:04:48 am] | Deleting remote file: /images/postdownloadscripts/fog.postdownload
[07-09-18 7:04:48 am] | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.PH-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.60.219.12
[07-09-18 7:04:48 am] * Started sync for Image postdownloadscripts
[07-09-18 7:04:48 am] | Replicating postinitscripts
[07-09-18 7:04:48 am] * Found Image to transfer to 2 s
[07-09-18 7:04:48 am] | File Name: dev/postinitscripts
[07-09-18 7:04:57 am] | dev/postinitscripts: No need to sync fog.postinit file to OC-FOG-NODE
[07-09-18 7:04:57 am] * All files synced for this item.
[07-09-18 7:05:06 am] | dev/postinitscripts: No need to sync fog.postinit file to PH-FOG-NODE
[07-09-18 7:05:06 am] * All files synced for this item.
[07-09-18 7:05:06 am] * Not syncing Image between s
[07-09-18 7:05:06 am] | Image Name: Win10BaseVM
[07-09-18 7:05:06 am] | There are no other members to sync to.
[07-09-18 7:05:06 am] * Not syncing Image between s
[07-09-18 7:05:06 am] | Image Name: Win7BaseVM
[07-09-18 7:05:06 am] | There are no other members to sync to.
[07-09-18 7:05:06 am] * Attempting to perform Group -> Nodes image replication.
[07-09-18 7:05:07 am] * Found Image to transfer to 2 s
[07-09-18 7:05:07 am] | Image Name: Win10BaseVM
[07-09-18 7:05:16 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE
[07-09-18 7:05:17 am] | Win10BaseVM: No need to sync d1.mbr file to OC-FOG-NODE
[07-09-18 7:05:17 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE
[07-09-18 7:05:17 am] | Win10BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE
[07-09-18 7:05:17 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE
[07-09-18 7:05:18 am] | Win10BaseVM: No need to sync d1.original.uuids file to OC-FOG-NODE
[07-09-18 7:05:18 am] | Win10BaseVM: No need to sync d1.partitions file to OC-FOG-NODE
[07-09-18 7:05:18 am] | Win10BaseVM: No need to sync d1p1.img file to OC-FOG-NODE
[07-09-18 7:05:19 am] | Win10BaseVM: No need to sync d1p2.img file to OC-FOG-NODE
[07-09-18 7:05:19 am] | Win10BaseVM: No need to sync d1p3.img file to OC-FOG-NODE
[07-09-18 7:07:17 am] | Files do not match on server: OC-FOG-NODE
[07-09-18 7:07:17 am] | Deleting remote file: /images/Win10BaseVM/d1p4.img
[07-09-18 7:07:17 am] | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win10BaseVM.transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win10BaseVM" "/images/Win10BaseVM"; exit' -u fog,[Protected] 10.59.181.12
[07-09-18 7:07:17 am] * Started sync for Image Win10BaseVM
[07-09-18 7:07:26 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE
[07-09-18 7:07:26 am] | Win10BaseVM: No need to sync d1.mbr file to PH-FOG-NODE
[07-09-18 7:07:27 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to PH-FOG-NODE
[07-09-18 7:07:27 am] | Win10BaseVM: No need to sync d1.original.fstypes file to PH-FOG-NODE
[07-09-18 7:07:27 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to PH-FOG-NODE
[07-09-18 7:07:27 am] | Win10BaseVM: No need to sync d1.original.uuids file to PH-FOG-NODE
[07-09-18 7:07:28 am] | Win10BaseVM: No need to sync d1.partitions file to PH-FOG-NODE
[07-09-18 7:07:28 am] | Win10BaseVM: No need to sync d1p1.img file to PH-FOG-NODE
[07-09-18 7:07:28 am] | Win10BaseVM: No need to sync d1p2.img file to PH-FOG-NODE
[07-09-18 7:07:29 am] | Win10BaseVM: No need to sync d1p3.img file to PH-FOG-NODE
[07-09-18 7:09:28 am] | Win10BaseVM: No need to sync d1p4.img file to PH-FOG-NODE
[07-09-18 7:09:28 am] * All files synced for this item.
[07-09-18 7:09:28 am] * Found Image to transfer to 2 s
[07-09-18 7:09:28 am] | Image Name: Win7BaseVM
[07-09-18 7:09:37 am] | Win7BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE
[07-09-18 7:09:37 am] | Win7BaseVM: No need to sync d1.mbr file to OC-FOG-NODE
[07-09-18 7:09:37 am] | Win7BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE
[07-09-18 7:09:38 am] | Win7BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE
[07-09-18 7:09:38 am] | Win7BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE
[07-09-18 7:09:38 am] | Win7BaseVM: No need to sync d1.partitions file to OC-FOG-NODE
[07-09-18 7:11:56 am] | Win7BaseVM: No need to sync d1p1.img file to OC-FOG-NODE
[07-09-18 7:11:56 am] * All files synced for this item.
[07-09-18 7:12:05 am] | Win7BaseVM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE
[07-09-18 7:12:05 am] | Win7BaseVM: No need to sync d1.mbr file to PH-FOG-NODE
[07-09-18 7:12:05 am] | Win7BaseVM: No need to sync d1.minimum.partitions file to PH-FOG-NODE
[07-09-18 7:12:06 am] | Win7BaseVM: No need to sync d1.original.fstypes file to PH-FOG-NODE
[07-09-18 7:12:06 am] | Win7BaseVM: No need to sync d1.original.swapuuids file to PH-FOG-NODE
[07-09-18 7:12:06 am] | Win7BaseVM: No need to sync d1.partitions file to PH-FOG-NODE
[07-09-18 7:14:21 am] | Win7BaseVM: No need to sync d1p1.img file to PH-FOG-NODE
[07-09-18 7:14:21 am] * All files synced for this item.
[07-09-18 7:29:21 am] * Starting Image Replication.
[07-09-18 7:29:21 am] * We are group ID: 1. We are group name: default
[07-09-18 7:29:21 am] * We are node ID: 1. We are node name: DO-FOG-MASTER
[07-09-18 7:29:21 am] * Attempting to perform Group -> Group image replication.
[07-09-18 7:29:21 am] | Replicating postdownloadscripts
[07-09-18 7:29:21 am] * Found Image to transfer to 2 s
[07-09-18 7:29:21 am] | File Name: postdownloadscripts
[07-09-18 7:29:30 am] | postdownloadscripts: No need to sync fog.deletelog file to OC-FOG-NODE
[07-09-18 7:29:31 am] | postdownloadscripts: No need to sync fog.drivers file to OC-FOG-NODE
[07-09-18 7:29:31 am] | postdownloadscripts: No need to sync fog.postdownload file to OC-FOG-NODE
[07-09-18 7:29:31 am] * All files synced for this item.
[07-09-18 7:29:40 am] | postdownloadscripts: No need to sync fog.deletelog file to PH-FOG-NODE
[07-09-18 7:29:41 am] | postdownloadscripts: No need to sync fog.drivers file to PH-FOG-NODE
[07-09-18 7:29:41 am] | postdownloadscripts: No need to sync fog.postdownload file to PH-FOG-NODE
[07-09-18 7:29:41 am] * All files synced for this item.
[07-09-18 7:29:41 am] | Replicating postinitscripts
[07-09-18 7:29:41 am] * Found Image to transfer to 2 s
[07-09-18 7:29:41 am] | File Name: dev/postinitscripts
[07-09-18 7:29:51 am] | dev/postinitscripts: No need to sync fog.postinit file to OC-FOG-NODE
[07-09-18 7:29:51 am] * All files synced for this item.
[07-09-18 7:30:00 am] | dev/postinitscripts: No need to sync fog.postinit file to PH-FOG-NODE
[07-09-18 7:30:00 am] * All files synced for this item.
[07-09-18 7:30:00 am] * Not syncing Image between s
[07-09-18 7:30:00 am] | Image Name: Win10BaseVM
[07-09-18 7:30:00 am] | There are no other members to sync to.
[07-09-18 7:30:00 am] * Not syncing Image between s
[07-09-18 7:30:00 am] | Image Name: Win7BaseVM
[07-09-18 7:30:00 am] | There are no other members to sync to.
[07-09-18 7:30:00 am] * Attempting to perform Group -> Nodes image replication.
[07-09-18 7:30:00 am] * Found Image to transfer to 2 s
[07-09-18 7:30:00 am] | Image Name: Win10BaseVM
[07-09-18 7:30:10 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE
[07-09-18 7:30:10 am] | Win10BaseVM: No need to sync d1.mbr file to OC-FOG-NODE
[07-09-18 7:30:10 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE
[07-09-18 7:30:11 am] | Win10BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE
[07-09-18 7:30:11 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE
[07-09-18 7:30:11 am] | Win10BaseVM: No need to sync d1.original.uuids file to OC-FOG-NODE
[07-09-18 7:30:11 am] | Win10BaseVM: No need to sync d1.partitions file to OC-FOG-NODE
[07-09-18 7:30:12 am] | Win10BaseVM: No need to sync d1p1.img file to OC-FOG-NODE
[07-09-18 7:30:12 am] | Win10BaseVM: No need to sync d1p2.img file to OC-FOG-NODE
[07-09-18 7:30:13 am] | Win10BaseVM: No need to sync d1p3.img file to OC-FOG-NODE
[07-09-18 7:32:18 am] | Files do not match on server: OC-FOG-NODE
[07-09-18 7:32:18 am] | Deleting remote file: /images/Win10BaseVM/d1p4.img
[07-09-18 7:32:18 am] | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win10BaseVM.transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win10BaseVM" "/images/Win10BaseVM"; exit' -u fog,[Protected] 10.59.181.12
[07-09-18 7:32:18 am] * Started sync for Image Win10BaseVM
[07-09-18 7:32:28 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE
[07-09-18 7:32:28 am] | Win10BaseVM: No need to sync d1.mbr file to PH-FOG-NODE
[07-09-18 7:32:28 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to PH-FOG-NODE
[07-09-18 7:32:29 am] | Win10BaseVM: No need to sync d1.original.fstypes file to PH-FOG-NODE
[07-09-18 7:32:29 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to PH-FOG-NODE
[07-09-18 7:32:29 am] | Win10BaseVM: No need to sync d1.original.uuids file to PH-FOG-NODE
[07-09-18 7:32:29 am] | Win10BaseVM: No need to sync d1.partitions file to PH-FOG-NODE
[07-09-18 7:32:30 am] | Win10BaseVM: No need to sync d1p1.img file to PH-FOG-NODE
[07-09-18 7:32:30 am] | Win10BaseVM: No need to sync d1p2.img file to PH-FOG-NODE
[07-09-18 7:32:31 am] | Win10BaseVM: No need to sync d1p3.img file to PH-FOG-NODE```

JGallo

@jflippen I to updated and noticed it is no longer in a replication loop. Interesting as I observed the replication log go through all my storage nodes is that first storage node has same file that does not match and one other storage node. So i’m waiting for next round of replication to occur and continue to observe. Update seem to fix the processes that @Tom-Elliott was talking about earlier and seems to check other storage nodes as replication goes down the line of storage nodes.

JGallo

I got a undefined variable error in replication log at the bottom. As you can see, replication is working but still states that d1p2.img file doesn’t match. Also, the file for the storage node with the error was perfectly fine on an earlier check. I didn’t update any images today.

[07-09-18 5:51:14 am]  | There are no other members to sync to.
[07-09-18 5:51:17 am]  * Found Image to transfer to 8 s
[07-09-18 5:51:17 am]  | Image Name: 32-Dell-790
[07-09-18 5:51:18 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to BCS-Master
[07-09-18 5:51:18 am]  | 32-Dell-790: No need to sync d1.mbr file to BCS-Master
[07-09-18 5:51:19 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to BCS-Master
[07-09-18 5:51:20 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to BCS-Master
[07-09-18 5:51:21 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to BCS-Master
[07-09-18 5:51:21 am]  | 32-Dell-790: No need to sync d1.partitions file to BCS-Master
[07-09-18 5:51:23 am]  | 32-Dell-790: No need to sync d1p1.img file to BCS-Master
[07-09-18 5:58:23 am]  | 32-Dell-790: No need to sync d1p2.img file to BCS-Master
[07-09-18 5:58:23 am]  * All files synced for this item.
[07-09-18 5:58:24 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to CHS-Master
[07-09-18 5:58:24 am]  | 32-Dell-790: No need to sync d1.mbr file to CHS-Master
[07-09-18 5:58:25 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to CHS-Master
[07-09-18 5:58:25 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to CHS-Master
[07-09-18 5:58:26 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to CHS-Master
[07-09-18 5:58:26 am]  | 32-Dell-790: No need to sync d1.partitions file to CHS-Master
[07-09-18 5:58:26 am]  | 32-Dell-790: No need to sync d1p1.img file to CHS-Master
[07-09-18 6:03:39 am]  | 32-Dell-790: No need to sync d1p2.img file to CHS-Master
[07-09-18 6:03:39 am]  * All files synced for this item.
[07-09-18 6:03:39 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to ESC-Master
[07-09-18 6:03:40 am]  | 32-Dell-790: No need to sync d1.mbr file to ESC-Master
[07-09-18 6:03:40 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to ESC-Master
[07-09-18 6:03:40 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to ESC-Master
[07-09-18 6:03:41 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to ESC-Master
[07-09-18 6:03:41 am]  | 32-Dell-790: No need to sync d1.partitions file to ESC-Master
[07-09-18 6:03:41 am]  | 32-Dell-790: No need to sync d1p1.img file to ESC-Master
[07-09-18 6:07:44 am]  | 32-Dell-790: No need to sync d1p2.img file to ESC-Master
[07-09-18 6:07:44 am]  * All files synced for this item.
[07-09-18 6:07:45 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to Jep-Master
[07-09-18 6:07:45 am]  | 32-Dell-790: No need to sync d1.mbr file to Jep-Master
[07-09-18 6:07:46 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to Jep-Master
[07-09-18 6:07:46 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to Jep-Master
[07-09-18 6:07:47 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to Jep-Master
[07-09-18 6:07:47 am]  | 32-Dell-790: No need to sync d1.partitions file to Jep-Master
[07-09-18 6:07:48 am]  | 32-Dell-790: No need to sync d1p1.img file to Jep-Master
[07-09-18 6:13:05 am]  | 32-Dell-790: No need to sync d1p2.img file to Jep-Master
[07-09-18 6:13:05 am]  * All files synced for this item.
[07-09-18 6:13:06 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VHS-Master
[07-09-18 6:13:07 am]  | 32-Dell-790: No need to sync d1.mbr file to VHS-Master
[07-09-18 6:13:07 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to VHS-Master
[07-09-18 6:13:08 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to VHS-Master
[07-09-18 6:13:08 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to VHS-Master
[07-09-18 6:13:09 am]  | 32-Dell-790: No need to sync d1.partitions file to VHS-Master
[07-09-18 6:13:10 am]  | 32-Dell-790: No need to sync d1p1.img file to VHS-Master
[07-09-18 6:19:08 am]  | 32-Dell-790: No need to sync d1p2.img file to VHS-Master
[07-09-18 6:19:08 am]  * All files synced for this item.
[07-09-18 6:19:10 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VP-Master
[07-09-18 6:19:11 am]  | 32-Dell-790: No need to sync d1.mbr file to VP-Master
[07-09-18 6:19:11 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to VP-Master
[07-09-18 6:19:12 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to VP-Master
[07-09-18 6:19:13 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to VP-Master
[07-09-18 6:19:13 am]  | 32-Dell-790: No need to sync d1.partitions file to VP-Master
[07-09-18 6:19:14 am]  | 32-Dell-790: No need to sync d1p1.img file to VP-Master
[07-09-18 6:28:00 am]  | 32-Dell-790: No need to sync d1p2.img file to VP-Master
[07-09-18 6:28:00 am]  * All files synced for this item.
[07-09-18 6:28:01 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to WCW-Master
[07-09-18 6:28:02 am]  | 32-Dell-790: No need to sync d1.mbr file to WCW-Master
[07-09-18 6:28:02 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to WCW-Master
[07-09-18 6:28:03 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to WCW-Master
[07-09-18 6:28:03 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to WCW-Master
[07-09-18 6:28:04 am]  | 32-Dell-790: No need to sync d1.partitions file to WCW-Master
[07-09-18 6:28:05 am]  | 32-Dell-790: No need to sync d1p1.img file to WCW-Master
[07-09-18 6:34:02 am]  | 32-Dell-790: No need to sync d1p2.img file to WCW-Master
[07-09-18 6:34:02 am]  * All files synced for this item.
[07-09-18 6:34:04 am]  * Found Image to transfer to 8 s
[07-09-18 6:34:04 am]  | Image Name: 64-Dell-790
[07-09-18 6:34:06 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to BCS-Master
[07-09-18 6:34:06 am]  | 64-Dell-790: No need to sync d1.mbr file to BCS-Master
[07-09-18 6:34:07 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to BCS-Master
[07-09-18 6:34:08 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to BCS-Master
[07-09-18 6:34:08 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to BCS-Master
[07-09-18 6:34:09 am]  | 64-Dell-790: No need to sync d1.partitions file to BCS-Master
[07-09-18 6:34:10 am]  | 64-Dell-790: No need to sync d1p1.img file to BCS-Master
[07-09-18 6:41:33 am]  | Files do not match on server: BCS-Master
[07-09-18 6:41:33 am]  | Deleting remote file: /images/64-Dell-790/d1p2.img
[07-09-18 6:41:33 am]  * Starting Sync Actions
[07-09-18 6:41:33 am]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.64-Dell-790.transfer.BCS-Master.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/64-Dell-790" "/images/64-Dell-790"; exit' -u fog,[Protected] 10.210.100.61
[07-09-18 6:41:33 am]  * Started sync for Image 64-Dell-790
[07-09-18 6:41:34 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to CHS-Master
[07-09-18 6:41:34 am]  | 64-Dell-790: No need to sync d1.mbr file to CHS-Master
[07-09-18 6:41:34 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to CHS-Master
[07-09-18 6:41:35 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to CHS-Master
[07-09-18 6:41:35 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to CHS-Master
[07-09-18 6:41:36 am]  | 64-Dell-790: No need to sync d1.partitions file to CHS-Master
[07-09-18 6:41:36 am]  | 64-Dell-790: No need to sync d1p1.img file to CHS-Master
[07-09-18 6:47:24 am]  | 64-Dell-790: No need to sync d1p2.img file to CHS-Master
[07-09-18 6:47:24 am]  * All files synced for this item.
[07-09-18 6:47:25 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to ESC-Master
[07-09-18 6:47:25 am]  | 64-Dell-790: No need to sync d1.mbr file to ESC-Master
[07-09-18 6:47:25 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to ESC-Master
[07-09-18 6:47:25 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to ESC-Master
[07-09-18 6:47:26 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to ESC-Master
[07-09-18 6:47:26 am]  | 64-Dell-790: No need to sync d1.partitions file to ESC-Master
[07-09-18 6:47:26 am]  | 64-Dell-790: No need to sync d1p1.img file to ESC-Master
[07-09-18 6:53:33 am]  | 64-Dell-790: No need to sync d1p2.img file to ESC-Master
[07-09-18 6:53:33 am]  * All files synced for this item.
[07-09-18 6:53:34 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to Jep-Master
[07-09-18 6:53:34 am]  | 64-Dell-790: No need to sync d1.mbr file to Jep-Master
[07-09-18 6:53:34 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to Jep-Master
[07-09-18 6:53:35 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to Jep-Master
[07-09-18 6:53:35 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to Jep-Master
[07-09-18 6:53:36 am]  | 64-Dell-790: No need to sync d1.partitions file to Jep-Master
[07-09-18 6:53:36 am]  | 64-Dell-790: No need to sync d1p1.img file to Jep-Master
[07-09-18 6:59:23 am]  | 64-Dell-790: No need to sync d1p2.img file to Jep-Master
[07-09-18 6:59:23 am]  * All files synced for this item.
[07-09-18 6:59:24 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to VHS-Master
[07-09-18 6:59:24 am]  | 64-Dell-790: No need to sync d1.mbr file to VHS-Master
[07-09-18 6:59:25 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to VHS-Master
[07-09-18 6:59:25 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to VHS-Master
[07-09-18 6:59:26 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to VHS-Master
[07-09-18 6:59:26 am]  | 64-Dell-790: No need to sync d1.partitions file to VHS-Master
[07-09-18 6:59:27 am]  | 64-Dell-790: No need to sync d1p1.img file to VHS-Master
[07-09-18 7:05:52 am]  | 64-Dell-790: No need to sync d1p2.img file to VHS-Master
[07-09-18 7:05:52 am]  * All files synced for this item.
[07-09-18 7:05:53 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to VP-Master
[07-09-18 7:05:54 am]  | 64-Dell-790: No need to sync d1.mbr file to VP-Master
[07-09-18 7:05:55 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to VP-Master
[07-09-18 7:05:55 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to VP-Master
[07-09-18 7:05:56 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to VP-Master
[07-09-18 7:05:57 am]  | 64-Dell-790: No need to sync d1.partitions file to VP-Master
[07-09-18 7:05:58 am]  | 64-Dell-790: No need to sync d1p1.img file to VP-Master
[07-09-18 7:13:12 am]  | Files do not match on server: VP-Master
[07-09-18 7:13:12 am]  | Deleting remote file: /images/64-Dell-790/d1p2.img
[07-09-18 7:13:12 am]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.64-Dell-790.transfer.VP-Master.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/64-Dell-790" "/images/64-Dell-790"; exit' -u fog,[Protected] 10.205.100.61
[07-09-18 7:13:12 am]  * Started sync for Image 64-Dell-790
[07-09-18 7:13:13 am]  | 64-Dell-790: No need to sync d1.fixed_size_partitions file to WCW-Master
[07-09-18 7:13:13 am]  | 64-Dell-790: No need to sync d1.mbr file to WCW-Master
[07-09-18 7:13:14 am]  | 64-Dell-790: No need to sync d1.minimum.partitions file to WCW-Master
[07-09-18 7:13:14 am]  | 64-Dell-790: No need to sync d1.original.fstypes file to WCW-Master
[07-09-18 7:13:15 am]  | 64-Dell-790: No need to sync d1.original.swapuuids file to WCW-Master
[07-09-18 7:13:15 am]  | 64-Dell-790: No need to sync d1.partitions file to WCW-Master
[07-09-18 7:13:16 am]  | 64-Dell-790: No need to sync d1p1.img file to WCW-Master
[07-09-18 7:19:33 am]  | 64-Dell-790: No need to sync d1p2.img file to WCW-Master
[07-09-18 7:19:33 am]  * All files synced for this item.
[07-09-18 7:19:33 am]  * Attempting to perform Group -> Nodes image replication.
[07-09-18 7:19:33 am]  * Not syncing Image between s
[07-09-18 7:19:33 am]  | Image Name: 32-Dell-790
[07-09-18 7:19:33 am]  | There are no other members to sync to.
[07-09-18 7:19:34 am]  * Not syncing Image between s
[07-09-18 7:19:34 am]  | Image Name: 64-Dell-790
[07-09-18 7:19:34 am]  | There are no other members to sync to.
[07-09-18 9:19:36 am]  * Starting Image Replication.
[07-09-18 9:19:36 am]  * We are group ID: 1. We are group name: default
[07-09-18 9:19:36 am]  * We are node ID: 1. We are node name: DefaultMember
[07-09-18 9:19:36 am]  * Attempting to perform Group -> Group image replication.
[07-09-18 9:19:36 am]  | Replicating postdownloadscripts
[07-09-18 9:19:37 am]  * Not syncing Image between s
[07-09-18 9:19:37 am]  | Image Name:
[07-09-18 9:19:37 am]  | There are no other members to sync to.
[07-09-18 9:19:37 am]  | Replicating postinitscripts
[07-09-18 9:19:37 am]  * Not syncing Image between s
[07-09-18 9:19:37 am]  | Image Name:
[07-09-18 9:19:37 am]  | There are no other members to sync to.
[07-09-18 9:19:40 am]  * Found Image to transfer to 8 s
[07-09-18 9:19:40 am]  | Image Name: 32-Dell-790
[07-09-18 9:19:41 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to BCS-Master
[07-09-18 9:19:42 am]  | 32-Dell-790: No need to sync d1.mbr file to BCS-Master
[07-09-18 9:19:43 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to BCS-Master
[07-09-18 9:19:43 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to BCS-Master
[07-09-18 9:19:44 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to BCS-Master
[07-09-18 9:19:45 am]  | 32-Dell-790: No need to sync d1.partitions file to BCS-Master
[07-09-18 9:19:45 am]  | 32-Dell-790: No need to sync d1p1.img file to BCS-Master
[07-09-18 9:26:50 am]  | 32-Dell-790: No need to sync d1p2.img file to BCS-Master
[07-09-18 9:26:50 am]  * All files synced for this item.
[07-09-18 9:26:50 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to CHS-Master
[07-09-18 9:26:51 am]  | 32-Dell-790: No need to sync d1.mbr file to CHS-Master
[07-09-18 9:26:51 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to CHS-Master
[07-09-18 9:26:51 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to CHS-Master
[07-09-18 9:26:52 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to CHS-Master
[07-09-18 9:26:52 am]  | 32-Dell-790: No need to sync d1.partitions file to CHS-Master
[07-09-18 9:26:53 am]  | 32-Dell-790: No need to sync d1p1.img file to CHS-Master
[07-09-18 9:32:07 am]  | 32-Dell-790: No need to sync d1p2.img file to CHS-Master
[07-09-18 9:32:07 am]  * All files synced for this item.
[07-09-18 9:32:08 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to ESC-Master
[07-09-18 9:32:08 am]  | 32-Dell-790: No need to sync d1.mbr file to ESC-Master
[07-09-18 9:32:08 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to ESC-Master
[07-09-18 9:32:09 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to ESC-Master
[07-09-18 9:32:09 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to ESC-Master
[07-09-18 9:32:09 am]  | 32-Dell-790: No need to sync d1.partitions file to ESC-Master
[07-09-18 9:32:10 am]  | 32-Dell-790: No need to sync d1p1.img file to ESC-Master
[07-09-18 9:36:35 am]  | 32-Dell-790: No need to sync d1p2.img file to ESC-Master
[07-09-18 9:36:35 am]  * All files synced for this item.
[07-09-18 9:36:36 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to Jep-Master
[07-09-18 9:36:36 am]  | 32-Dell-790: No need to sync d1.mbr file to Jep-Master
[07-09-18 9:36:37 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to Jep-Master
[07-09-18 9:36:37 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to Jep-Master
[07-09-18 9:36:37 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to Jep-Master
[07-09-18 9:36:38 am]  | 32-Dell-790: No need to sync d1.partitions file to Jep-Master
[07-09-18 9:36:38 am]  | 32-Dell-790: No need to sync d1p1.img file to Jep-Master
[07-09-18 9:41:56 am]  | 32-Dell-790: No need to sync d1p2.img file to Jep-Master
[07-09-18 9:41:56 am]  * All files synced for this item.
[07-09-18 9:41:58 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VHS-Master
[07-09-18 9:41:59 am]  | 32-Dell-790: No need to sync d1.mbr file to VHS-Master
[07-09-18 9:42:00 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to VHS-Master
[07-09-18 9:42:00 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to VHS-Master
[07-09-18 9:42:01 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to VHS-Master
[07-09-18 9:42:01 am]  | 32-Dell-790: No need to sync d1.partitions file to VHS-Master
[07-09-18 9:42:02 am]  | 32-Dell-790: No need to sync d1p1.img file to VHS-Master
[07-09-18 9:49:02 am]  | 32-Dell-790: No need to sync d1p2.img file to VHS-Master
[07-09-18 9:49:02 am]  * All files synced for this item.
[07-09-18 9:49:03 am]  | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VP-Master
[07-09-18 9:49:04 am]  | 32-Dell-790: No need to sync d1.mbr file to VP-Master
[07-09-18 9:49:05 am]  | 32-Dell-790: No need to sync d1.minimum.partitions file to VP-Master
[07-09-18 9:49:05 am]  | 32-Dell-790: No need to sync d1.original.fstypes file to VP-Master
[07-09-18 9:49:06 am]  | 32-Dell-790: No need to sync d1.original.swapuuids file to VP-Master
[07-09-18 9:49:07 am]  | 32-Dell-790: No need to sync d1.partitions file to VP-Master
[07-09-18 9:49:08 am]  | 32-Dell-790: No need to sync d1p1.img file to VP-Master
[07-09-18 9:56:38 am]  | Files do not match on server: VP-Master
[07-09-18 9:56:38 am]  | Deleting remote file: /images/32-Dell-790/d1p2.img
[07-09-18 9:56:38 am]  | CMD:
                        lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.32-Dell-790.transfer.VP-Master.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/32-Dell-790" "/images/32-Dell-790"; exit' -u fog,[Protected] 10.205.100.61
[07-09-18 9:56:38 am]  * Started sync for Image 32-Dell-790
[07-09-18 9:56:38 am]  * Type: 8, File: /var/www/fog/lib/service/fogservice.class.php, Line: 481, Message: Undefined variable: itemType, Host: 10.225.100.61, Username: fog

jflippen

@jgallo @Tom-Elliott
In another thread I was looking into the code a bit and Tom verified that the following code compares the files with one being hashed on one server and the other being hashed on the other.

private static function _filesAreEqual($size_a, $size_b, $file_a, $file_b, $avail) {
        if ($size_a != $size_b) {
            return false;
        }
        if (false === $avail) {
            if ($size_a < 1047685760) {
                $remhash = md5_file($file_b);
                $lochash = md5_file($file_a);
                return ($remhash == $lochash);
            }
            return file_exists($file_b) && file_exists($file_a);
        }
        $hashLoc = self::getHash($file_a);
        $hashRem = $file_b;
        $hashCom = ($hashLoc == $hashRem);
        return $hashCom;

Tom, is it possible that the function is checking for the $hashRem variable before the other server finishes it’s md5hash and therefore comes up with a mismatch? Still doesn’t explain why it won’t delete the file and replace though…

george1421

@jflippen Just as an idea (first let me say I’m not a programmer), if you look about in the code where you can find an example of the replication agent writing to a log file. Clone that and place it in the correct location in the code to write both md5 hash codes into the log. Once the fog server has restarted then it should log that information into the replicator log file. I’ve had to do somethings similar in the past to reverse engineer some of the magic Tom does with his code.

Wayne Workman

@jflippen said in Replication problems 1.5.4 - always copying:

is it possible that the function is checking for the $hashRem variable before the other server finishes it’s md5hash and therefore comes up with a mismatch?

Those calls are blocking-type calls, the code doesn’t move to the next line until they are completed or time out.

jflippen

@wayne-workman Thanks Wayne. Is it possible that the block is timing out then if the file is taking too long on the remote server to create the md5sum and that is why it keeps trying to replace the same file each time there is a replication cycle? Is there a place I can change the time it takes to timeout to try as a troubleshooting technique (or I might go with George’s technique and try and echo out the two variables to a log during that pass to check and see if it is truly grabbing the $hashRem

Tom Elliott

@jflippen PHP is a blocking methodology by its very nature. So it couldn’t continue moving forward if one part is “stuck”. (Unless of course it’s reaching maximum execution timeout, which shouldn’t happen but I don’t know everyone’s environment.)

Tom Elliott

Sorry Read the post and replied, didn’t realize the message had been replied too already.

Sebastian Roth

@ablohowiak @jflippen @Hanz While trying to figure out some replication hickup I found and fixed a couple of issues within the code. All that will be in the next release. Hopefully coming soon. Let me know if you are keen to test those changes beforehand.

jflippen

@Sebastian-Roth Definitely! Though I will probably test on some VM’s first since our actual FOG server is running scripts to back everything up to our nodes instead since we were having issues with both replication services. Will the test code be on the -working channel or a different one? I’ll try to provide feedback when I can. There are other bugs I need to report on as well, but haven’t had time to document.

Sebastian Roth

@jflippen The current changes are on a new branch replication (link) which I will merge into working after a first round of feedback.

Important notice: I had to change some of the hashing code too and therefore nodes being on different versions (1.5.4 or working VS. replication branch) will end up replicating images over and over again. So you need to have all nodes on the replication branch or setup up a separate test environment!!

Please make sure you stop replication first (systemctl stop FOGImageReplicator), then update the storage node and after that update master node.

jflippen

@Sebastian-Roth Okay, so I managed to test out the new branch at home on my VM’s (like I mentioned before, not going to touch our primary FOG server right now sense we have a work around). Here is what I found:

The first time around I tried just doing an in-place upgrade of the working 1.6 branch. This, however seemed to not work properly as it did not change anything after I swapped branches and installed on all three (stopping the service and rebooting on each VM, going from nodes to master).
The 2nd time I did a fresh install of Centos 7 on each VM, followed the wiki guide, and went directly to the replication branch. I then created a fake image by creating a folder and then putting some random files in there (I don’t have PXE boot or a server environment at home). When I checked the log and did some testing, it looks like it’s working! Previously, FOG would not delete mismatched files, meaning the files would never get updated on the node, but would keep trying to run an lftp sync… It would even say files didn’t match even if they had the same md5sum at times. This seems to have been fixed in your new branch! I have included my log below.
During testing I noticed that the replication service won’t remove files that are on the node but not the master (from changing the files I was testing with on the master). Would it be possible to implement an if exist variable that checks an array of the file names in an image folder against the master? This would be handy for keeping the driver injection folder we use in the images folder synced properly too when removing deprecated models.

The Log, as promised

[11-07-18 6:34:23 am] 
==================================
===        ====    =====      ====
===  =========  ==  ===   ==   ===
===  ========  ====  ==  ====  ===
===  ========  ====  ==  =========
===      ====  ====  ==  =========
===  ========  ====  ==  ===   ===
===  ========  ====  ==  ====  ===
===  =========  ==  ===   ==   ===
===  ==========    =====      ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[11-07-18 6:34:23 am] Interface Ready with IP Address: 127.0.0.1
[11-07-18 6:34:23 am] Interface Ready with IP Address: 127.0.1.1
[11-07-18 6:34:23 am] Interface Ready with IP Address: 192.168.1.16
[11-07-18 6:34:23 am] Interface Ready with IP Address: FOG-MASTER
[11-07-18 6:34:23 am] FOGService: ImageReplicator - Waiting for mysql to be available
[11-07-18 6:34:35 am]  * Starting ImageReplicator Service
[11-07-18 6:34:35 am]  * Checking for new items every 600 seconds
[11-07-18 6:34:35 am]  * Starting service loop
[11-07-18 6:34:36 am]  * Starting Image Replication.
[11-07-18 6:34:36 am]  * We are group ID: 1. We are group name: default
[11-07-18 6:34:36 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 6:34:36 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 6:34:36 am]  | Replicating postdownloadscripts
[11-07-18 6:34:38 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:34:38 am]  | File Name: postdownloadscripts
[11-07-18 6:34:38 am]  | NODE-1 server does not appear to be online.
[11-07-18 6:34:38 am]  | NODE-2 server does not appear to be online.
[11-07-18 6:34:38 am]  | Replicating postinitscripts
[11-07-18 6:34:40 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:34:40 am]  | File Name: dev/postinitscripts
[11-07-18 6:34:40 am]  | NODE-1 server does not appear to be online.
[11-07-18 6:34:40 am]  | NODE-2 server does not appear to be online.
[11-07-18 6:34:40 am]  * Not syncing Image between groups
[11-07-18 6:34:40 am]  | Image Name: test
[11-07-18 6:34:40 am]  | There are no other members to sync to.
[11-07-18 6:34:40 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 6:34:42 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:34:42 am]  | Image Name: test
[11-07-18 6:34:42 am]  | NODE-1 server does not appear to be online.
[11-07-18 6:34:42 am]  | NODE-2 server does not appear to be online.
[11-07-18 6:44:42 am]  * Starting Image Replication.
[11-07-18 6:44:42 am]  * We are group ID: 1. We are group name: default
[11-07-18 6:44:42 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 6:44:42 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 6:44:42 am]  | Replicating postdownloadscripts
[11-07-18 6:44:42 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:44:42 am]  | File Name: postdownloadscripts
[11-07-18 6:44:43 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-07-18 6:44:43 am]  * All files synced for this item.
[11-07-18 6:44:43 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-07-18 6:44:43 am]  * All files synced for this item.
[11-07-18 6:44:43 am]  | Replicating postinitscripts
[11-07-18 6:44:44 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:44:44 am]  | File Name: dev/postinitscripts
[11-07-18 6:44:44 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-07-18 6:44:44 am]  * All files synced for this item.
[11-07-18 6:44:45 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-07-18 6:44:45 am]  * All files synced for this item.
[11-07-18 6:44:45 am]  * Not syncing Image between groups
[11-07-18 6:44:45 am]  | Image Name: test
[11-07-18 6:44:45 am]  | There are no other members to sync to.
[11-07-18 6:44:45 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 6:44:46 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:44:46 am]  | Image Name: test
[11-07-18 6:44:46 am]   # test: File does not exist 2014-10-31 11.52.32.jpg (NODE-1)
[11-07-18 6:44:46 am]   # test: File size mismatch - CentOS-7-x86_64-DVD-1804.iso: 4470079488 != 2903506944
[11-07-18 6:44:46 am]   # test: Deleting remote file CentOS-7-x86_64-DVD-1804.iso
[11-07-18 6:44:46 am]   # test: File does not exist Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-07-18 6:44:46 am]   # test: File does not exist FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-07-18 6:44:46 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.test.transfer.NODE-1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.17
[11-07-18 6:44:46 am]  | Started sync for Image test - Resource id #714
[11-07-18 6:44:46 am]   # test: File does not exist 2014-10-31 11.52.32.jpg (NODE-2)
[11-07-18 6:44:46 am]   # test: File size mismatch - CentOS-7-x86_64-DVD-1804.iso: 4470079488 != 2903506944
[11-07-18 6:44:46 am]   # test: Deleting remote file CentOS-7-x86_64-DVD-1804.iso
[11-07-18 6:44:46 am]   # test: File does not exist Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-07-18 6:44:46 am]   # test: File does not exist FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-07-18 6:44:46 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.test.transfer.NODE-2.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.18
[11-07-18 6:44:46 am]  | Started sync for Image test - Resource id #729
[11-07-18 6:48:10 am]  | Sync finished - Resource id #714
[11-07-18 6:48:11 am]  | Sync finished - Resource id #729
[11-07-18 6:54:42 am]  * Starting Image Replication.
[11-07-18 6:54:42 am]  * We are group ID: 1. We are group name: default
[11-07-18 6:54:42 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 6:54:42 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 6:54:42 am]  | Replicating postdownloadscripts
[11-07-18 6:54:43 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:54:43 am]  | File Name: postdownloadscripts
[11-07-18 6:54:43 am]   # postdownloadscripts: File does not exist fog.custominstall (NODE-1)
[11-07-18 6:54:43 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-07-18 6:54:43 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.17
[11-07-18 6:54:43 am]  | Started sync for Image postdownloadscripts - Resource id #781
[11-07-18 6:54:44 am]   # postdownloadscripts: File does not exist fog.custominstall (NODE-2)
[11-07-18 6:54:44 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-07-18 6:54:44 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-2.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.18
[11-07-18 6:54:44 am]  | Started sync for Image postdownloadscripts - Resource id #794
[11-07-18 6:54:44 am]  | Replicating postinitscripts
[11-07-18 6:54:45 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:54:45 am]  | File Name: dev/postinitscripts
[11-07-18 6:54:45 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-07-18 6:54:45 am]  * All files synced for this item.
[11-07-18 6:54:45 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-07-18 6:54:45 am]  * All files synced for this item.
[11-07-18 6:54:46 am]  * Not syncing Image between groups
[11-07-18 6:54:46 am]  | Image Name: test
[11-07-18 6:54:46 am]  | There are no other members to sync to.
[11-07-18 6:54:46 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 6:54:46 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:54:46 am]  | Image Name: test
[11-07-18 6:54:47 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-07-18 6:54:47 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-1)
[11-07-18 6:54:48 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-07-18 6:54:48 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-07-18 6:54:48 am]  * All files synced for this item.
[11-07-18 6:54:49 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-07-18 6:54:49 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-2)
[11-07-18 6:54:49 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-07-18 6:54:50 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-07-18 6:54:50 am]  * All files synced for this item.
[11-07-18 6:54:50 am]  | Sync finished - Resource id #781
[11-07-18 6:54:50 am]  | Sync finished - Resource id #794
[11-07-18 7:04:42 am]  * Starting Image Replication.
[11-07-18 7:04:42 am]  * We are group ID: 1. We are group name: default
[11-07-18 7:04:42 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 7:04:42 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 7:04:42 am]  | Replicating postdownloadscripts
[11-07-18 7:04:42 am]  * Found Image to transfer to 2 nodes
[11-07-18 7:04:42 am]  | File Name: postdownloadscripts
[11-07-18 7:04:42 am]   # postdownloadscripts: File does not exist fog.copydrivers (NODE-1)
[11-07-18 7:04:42 am]   # postdownloadscripts: File size mismatch - fog.custominstall: 1673 != 235
[11-07-18 7:04:42 am]   # postdownloadscripts: Deleting remote file fog.custominstall
[11-07-18 7:04:43 am]   # postdownloadscripts: File size mismatch - fog.postdownload: 271 != 235
[11-07-18 7:04:43 am]   # postdownloadscripts: Deleting remote file fog.postdownload
[11-07-18 7:04:43 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.17
[11-07-18 7:04:43 am]  | Started sync for Image postdownloadscripts - Resource id #975
[11-07-18 7:04:43 am]   # postdownloadscripts: File does not exist fog.copydrivers (NODE-2)
[11-07-18 7:04:43 am]   # postdownloadscripts: File size mismatch - fog.custominstall: 1673 != 235
[11-07-18 7:04:43 am]   # postdownloadscripts: Deleting remote file fog.custominstall
[11-07-18 7:04:43 am]   # postdownloadscripts: File size mismatch - fog.postdownload: 271 != 235
[11-07-18 7:04:43 am]   # postdownloadscripts: Deleting remote file fog.postdownload
[11-07-18 7:04:43 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-2.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.18
[11-07-18 7:04:43 am]  | Started sync for Image postdownloadscripts - Resource id #993
[11-07-18 7:04:43 am]  | Replicating postinitscripts
[11-07-18 7:04:44 am]  * Found Image to transfer to 2 nodes
[11-07-18 7:04:44 am]  | File Name: dev/postinitscripts
[11-07-18 7:04:44 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-07-18 7:04:44 am]  * All files synced for this item.
[11-07-18 7:04:44 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-07-18 7:04:44 am]  * All files synced for this item.
[11-07-18 7:04:45 am]  * Not syncing Image between groups
[11-07-18 7:04:45 am]  | Image Name: test
[11-07-18 7:04:45 am]  | There are no other members to sync to.
[11-07-18 7:04:45 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 7:04:45 am]  * Found Image to transfer to 2 nodes
[11-07-18 7:04:45 am]  | Image Name: test
[11-07-18 7:04:46 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-07-18 7:04:46 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-1)
[11-07-18 7:04:46 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-07-18 7:04:47 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-07-18 7:04:47 am]  * All files synced for this item.
[11-07-18 7:04:47 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-07-18 7:04:47 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-2)
[11-07-18 7:04:48 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-07-18 7:04:48 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-07-18 7:04:48 am]  * All files synced for this item.
[11-07-18 7:04:48 am]  | Sync finished - Resource id #975
[11-07-18 7:04:48 am]  | Sync finished - Resource id #993

Replication problems 1.5.4 - always copying

209

11.6k

17.1k

154.6k