Replication problems 1.5.4 - always copying
-
Ubuntu 16.04 and Centos 7, all nodes on 1.5.4
I’m having issues with the replication getting stuck on the same node.
Things step through copying postdownloadscripts: there’s no need to sync for a number of nodes but there is one that “needs it” every time even though the files are there. The problem is it never syncs with any other node past that one even if I wait for days. By restarting the FOGImageReplication service I’m able to observe the behavior repeat.
I also have the same problem with an image, bit it occurs a different node than the postdownloadscripts.
If I disable the problematic storage nodes the replication process will move on to the other nodes.
-
@ablohowiak I think I read the developers fixed this issue (or it was another replication issue) in the working branch (pre fog 1.5.5).
If you could test this for us?
These instructions assume that you downloaded the fog installer files via git.
- Change to the local git repo for fog. Depending on which instructions you followed they will be in /root/fogproject or /opt/fogproject.
- key in the following:
sudo git pull sudo git checkout working sudo git pull cd bin
Reinstall fog using the working branch
sudo ./installfog.sh
Then at the end so you don’t forget switch back.
cd /root/fogproject sudo git checkout master
-
I’ve installed the trunk version on the Master storage node. Is there a way to confirm the version on a storage node?
There’s still a replication issue. Seems that the d1p2.img is always the problematic file?
[07-02-18 9:18:03 am] | Files do not match on server: FogKennedy
[07-02-18 9:18:03 am] | Deleting remote file: /images/000-10-Golden/d1p2.img
[07-02-18 9:18:03 am] * Starting Sync Actions
[07-02-18 9:18:03 am] | CMD:
lftp -e ‘set xfer:log 1; set xfer:log-file “/opt/fog/log/fogreplicator.000-10-Golden.transfer.FogKennedy.log”;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude “.srvprivate” “/images/000-10-Golden” “/images/000-10-Golden”; exit’ -u fog,[Protected] 10.97.0.17
[07-02-18 9:18:03 am] * Started sync for Image 000-10-Golden
[07-02-18 9:18:03 am] | Replication already running with PID: 7307The log file says that’s it’s deleting the file and starting a sync. There is no change in the timestamp on the destination node image file and no transfer log file is being created on the master storage node.
-
@ablohowiak did the working branch end up helping with the replication? I’m currently having the same issue where image replication loop is occurring on the first storage node. My observation is that it only occurs after the initial upload meaning if you decide to update an existing image that has only been uploaded once “Original upload” and then upload the updated image, replication loops only on that image on the first storage node. Updating to working branch seemed like it worked but then eventually loops again. So i’m not sure if it’s completely resolved. I also have storage nodes and not a NAS system.
-
@george1421 Dear god I hope you’re right. I haven’t had a chance to upgrade to working branch 1.5.5 yet… been busy with other summer work. Oddly enough, I found my snapins update just fine when it comes to replication… it is just the images that never deleted the old files that didn’t match. I will try this soon as I get a chance.
-
@jgallo the working branch didn’t appear to fix the issue. I’m observing the exact same thing that you described. It makes adding a storage node to group, or a new image unreliable.
-
@ablohowiak Not sure I understand what you mean here.
Are the files not replicating, or not deleting? The node’s status, itself, should be fine.
-
https://forums.fogproject.org/topic/11381/image-not-syncing-between-nodes
I’ve been going through this thread and seems very identical to what we are experiencing. I think the only difference is that person only had one node as to maybe @ablohowiak and I having multiple storage nodes with various storage groups. rsync worked like a champ so rsync’ing the master nodes and replication to storage groups works fine also.
-
@tom-elliott Based off my last reply to you on this thread:
https://forums.fogproject.org/topic/12037/replication-only-working-on-1-5-nodesI have noticed that it has not been deleting the files when it says it will in the log. I can manually delete them via ftp and the fog username / password. Also, the snapins will update on the nodes just fine when I re-upload a snapin, so it just seems to be an issue with the images. I think what they are referring to is that since they have multiple nodes, as do I, the image never replicates to the other nodes because it keeps finding a fault and never updating the one of the first one in line.
-
I’m not replicating the snapins just images. Things worked okay when the storage group was small, and I wasn’t replicating images across storage groups.
Now as storage nodes are added the images sometimes get there, but usually things get hung up on a node not deleting and copying an image. The process loops before making it through all the storage nodes.
-
@ablohowiak Do you think this could be a permissions issue? I think someone mentioned that in a different post but here is my observation from original upload of an image to uploading an updated version of the original image:
permissions of folder on original image - 777 with user:group - fog:fog
permissions of folder after uploading updated image - 755 with user:group - fog:root
I’m currently manually setting permissions back to original settings as if it was a fresh upload then going to upload an updated version and re-check the permissions and see if changes occurred. What i’m hoping for: if changing the permissions of image folders to their original state, then I hope a replication loop doesn’t occur. If it does not, then i will go ahead and upload and continue to observe.
-
@jgallo 755 for fog:root should be fine permissions to allow delete. This is because 7 is the “user” permissions (which is read, write, execute). The first 5 is the group permissions which is read and execute. So the “root” permissions. The second 5 is the “other” which is also read and execute.
I’m not saying you didn’t know this already, just pointing out that the permissions being 777 or 755 should not matter, especially as capture doesn’t have a problem moving the files in place.
Of course any information might help lead to a more suitable solution, but this shouldn’t be a problem at all. It’s worrying as I don’t know where to begin troubleshooting this issue. Especially since the first replication process seems to work perfectly fine.
-
@tom-elliott No worries, I was just throwing some observations out there. My thought process was that since we are dealing with replication and storage groups with various master servers maybe somewhere down the line permissions could be not properly set since if the replication service doesn’t see the file consistency. As I am as lost as any other person with this issue, lol, i’m hoping that maybe some observation input could at least steer you in some direction to troubleshoot.
I just uploaded an image to a location that has a storage group defined and has two storage nodes within that group. So tailing the replication log it eventually shows that files need to replicate. Replication occurs but the log just sits there as if replication occurred to all the nodes. Eventually once replication service restarts from the sleeptime settings in FOG, replication starts fresh and eventually goes back to a loop to same files and same storage node. Hope that helps.
-
It doesn’t appear to be a permissions issue. I’m not seeing an entire image as the problem, usually just the largest file, but it’s inconsistent as to which image or storage node will find that the “Files do not match on server”.
[07-06-18 8:25:24 am] * All files synced for this item.
[07-06-18 8:25:25 am] | 000-10-Golden: No need to sync d1.fixed_size_partitions file to FogAllis
[07-06-18 8:25:26 am] | 000-10-Golden: No need to sync d1.mbr file to FogAllis
[07-06-18 8:25:27 am] | 000-10-Golden: No need to sync d1.minimum.partitions file to FogAllis
[07-06-18 8:25:28 am] | 000-10-Golden: No need to sync d1.original.fstypes file to FogAllis
[07-06-18 8:25:28 am] | 000-10-Golden: No need to sync d1.original.swapuuids file to FogAllis
[07-06-18 8:25:29 am] | 000-10-Golden: No need to sync d1.partitions file to FogAllis
[07-06-18 8:25:30 am] | 000-10-Golden: No need to sync d1p1.img file to FogAllis
[07-06-18 8:32:15 am] | Files do not match on server: FogAllis
[07-06-18 8:32:16 am] | Deleting remote file: /images/000-10-Golden/d1p2.img
[07-06-18 8:32:16 am] * Starting Sync Actions
[07-06-18 8:32:16 am] | CMD:
lftp -e 'set xfer:log 1; set xfer:log-file “/opt/fog/log/fogreplicator.000-10-Golden.transfer.FogAllis.log”drwxrwxrwx 16 fog root 4096 Jun 11 14:12 …
-rwxr-xr-x 1 fog fog 3 Jun 11 10:59 d1.fixed_size_partitions
-rwxr-xr-x 1 fog fog 1048576 Jun 11 10:59 d1.mbr
-rwxr-xr-x 1 fog fog 190 Jun 11 10:59 d1.minimum.partitions
-rwxr-xr-x 1 fog fog 15 Jun 11 10:59 d1.original.fstypes
-rwxr-xr-x 1 fog fog 0 Jun 11 10:59 d1.original.swapuuids
-rwxr-xr-x 1 fog fog 9118461 Jun 11 11:00 d1p1.img
-rwxr-xr-x 1 fog fog 13976945113 Jun 11 12:12 d1p2.img
-rwxr-xr-x 1 fog fog 190 Jun 11 10:59 d1.partitionsBut size isn’t the issue, because this is happening with the small files too.
[07-06-18 9:05:29 am] | Files do not match on server: 301Rack
[07-06-18 9:05:29 am] | Deleting remote file: /images/postdownloadscripts/fog.postdownload
[07-06-18 9:05:29 am] | postdownloadscripts: No need to sync fog.postdownload.orig file to 301Rack
[07-06-18 9:05:29 am] * Starting Sync Actions
[07-06-18 9:05:29 am] | CMD:
lftp -e ‘set xfer:log 1; set xfer:log-file “/opt/fog/log/fogreplicator…transfer.301Rack.log”;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude “.srvprivate” “/images/postdownloadscripts” “/images/postdownloadscripts”; exit’ -u fog,[Protected] 10.129.0.17
[07-06-18 9:05:29 am] * Started sync for Image postdownloadscripts
[07-06-18 9:05:29 am] | Replication already running with PID: 8510The file isn’t deleted, and no transfer log file is ever created. My nodes aren’t all connected at the same speed so I’m going to try throttling the replication to see if it has any impact.
-
@tom-elliott I can verify through multiple attempts that the images are being replicated to the first storage node, but then the replication stops completely. Every other attempt, it seems to want to replicate the biggest (main) partitions as you can see below in the log. d1p3 for UEFI or d1p2 for MBR images.
[07-07-18 12:19:26 pm] | There are no other members to sync to. [07-07-18 12:19:26 pm] | Image Name: SysprepWin10EDU [07-07-18 12:19:26 pm] * Not syncing Image between s [07-07-18 12:19:26 pm] | There are no other members to sync to. [07-07-18 12:19:26 pm] | Image Name: Sysprep-Win10EDU-X64 [07-07-18 12:19:26 pm] * Not syncing Image between s [07-07-18 12:19:25 pm] | There are no other members to sync to. [07-07-18 12:19:25 pm] | Image Name: Sysprep-UEFI-x64 [07-07-18 12:19:25 pm] * Not syncing Image between s [07-07-18 12:19:25 pm] | There are no other members to sync to. [07-07-18 12:19:25 pm] | Image Name: Sysprep-All (Win 7) [07-07-18 12:19:25 pm] * Not syncing Image between s [07-07-18 12:19:24 pm] * Attempting to perform Group -> Nodes image replication. [07-07-18 12:19:24 pm] | Replication already running with PID: 40445 [07-07-18 12:19:24 pm] | Replication already running with PID: 40445 [07-07-18 12:19:24 pm] | Replication already running with PID: 40445 [07-07-18 12:19:24 pm] | Replication already running with PID: 40445 [07-07-18 12:19:24 pm] | Replication already running with PID: 40445 [07-07-18 12:19:24 pm] | Replication already running with PID: 40445 [07-07-18 12:19:24 pm] | Replication already running with PID: 40445 [07-07-18 12:19:24 pm] * Started sync for Image SysprepWin10EDU lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.SysprepWin10EDU.transfer.Ashford.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepWin10EDU" "/images/SysprepWin10EDU"; exit' -u fog,[Protected] 10.72.128.55 [07-07-18 12:19:24 pm] | CMD: [07-07-18 12:19:24 pm] * Starting Sync Actions [07-07-18 12:19:24 pm] | Deleting remote file: /images/SysprepWin10EDU/d1p1.img [07-07-18 12:19:24 pm] | Files do not match on server: Ashford [07-07-18 12:17:57 pm] | SysprepWin10EDU: No need to sync d1.partitions file to Ashford [07-07-18 12:17:57 pm] | SysprepWin10EDU: No need to sync d1.original.swapuuids file to Ashford [07-07-18 12:17:56 pm] | SysprepWin10EDU: No need to sync d1.original.fstypes file to Ashford [07-07-18 12:17:56 pm] | SysprepWin10EDU: No need to sync d1.minimum.partitions file to Ashford [07-07-18 12:17:55 pm] | SysprepWin10EDU: No need to sync d1.mbr file to Ashford [07-07-18 12:17:54 pm] | SysprepWin10EDU: No need to sync d1.fixed_size_partitions file to Ashford [07-07-18 12:17:53 pm] | Image Name: SysprepWin10EDU [07-07-18 12:17:53 pm] * Found Image to transfer to 9 s [07-07-18 12:17:50 pm] | Replication already running with PID: 28731 [07-07-18 12:17:50 pm] | Replication already running with PID: 28731 [07-07-18 12:17:50 pm] | Replication already running with PID: 28731 [07-07-18 12:17:50 pm] | Replication already running with PID: 28731 [07-07-18 12:17:50 pm] | Replication already running with PID: 28731 [07-07-18 12:17:50 pm] | Replication already running with PID: 28731 [07-07-18 12:17:50 pm] | Replication already running with PID: 28731 [07-07-18 12:17:50 pm] * Started sync for Image Sysprep-Win10EDU-X64 lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-Win10EDU-X64.transfer.Ashford.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepImagex64" "/images/SysprepImagex64"; exit' -u fog,[Protected] 10.72.128.55 [07-07-18 12:17:50 pm] | CMD: [07-07-18 12:17:50 pm] * Starting Sync Actions [07-07-18 12:17:50 pm] | Deleting remote file: /images/SysprepImagex64/d1p2.img [07-07-18 12:17:50 pm] | Files do not match on server: Ashford [07-07-18 12:16:03 pm] | Sysprep-Win10EDU-X64: No need to sync d1p1.img file to Ashford [07-07-18 12:16:02 pm] | Sysprep-Win10EDU-X64: No need to sync d1.partitions file to Ashford [07-07-18 12:16:01 pm] | Sysprep-Win10EDU-X64: No need to sync d1.original.swapuuids file to Ashford [07-07-18 12:16:01 pm] | Sysprep-Win10EDU-X64: No need to sync d1.original.fstypes file to Ashford [07-07-18 12:16:00 pm] | Sysprep-Win10EDU-X64: No need to sync d1.minimum.partitions file to Ashford [07-07-18 12:16:00 pm] | Sysprep-Win10EDU-X64: No need to sync d1.mbr file to Ashford [07-07-18 12:15:59 pm] | Sysprep-Win10EDU-X64: No need to sync d1.fixed_size_partitions file to Ashford [07-07-18 12:15:58 pm] | Image Name: Sysprep-Win10EDU-X64 [07-07-18 12:15:58 pm] * Found Image to transfer to 9 s [07-07-18 12:15:55 pm] | Replication already running with PID: 14564 [07-07-18 12:15:55 pm] | Replication already running with PID: 14564 [07-07-18 12:15:55 pm] | Replication already running with PID: 14564 [07-07-18 12:15:55 pm] | Replication already running with PID: 14564 [07-07-18 12:15:55 pm] | Replication already running with PID: 14564 [07-07-18 12:15:55 pm] | Replication already running with PID: 14564 [07-07-18 12:15:55 pm] | Replication already running with PID: 14564 [07-07-18 12:15:55 pm] * Started sync for Image Sysprep-UEFI-x64 lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-UEFI-x64.transfer.Ashford.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Sysprep-UEFI-x64" "/images/Sysprep-UEFI-x64"; exit' -u fog,[Protected] 10.72.128.55 [07-07-18 12:15:55 pm] | CMD: [07-07-18 12:15:55 pm] * Starting Sync Actions [07-07-18 12:15:55 pm] | Sysprep-UEFI-x64: No need to sync d1p4.img file to Ashford [07-07-18 12:15:54 pm] | Deleting remote file: /images/Sysprep-UEFI-x64/d1p3.img [07-07-18 12:15:54 pm] | Files do not match on server: Ashford [07-07-18 12:13:49 pm] | Sysprep-UEFI-x64: No need to sync d1p2.img file to Ashford [07-07-18 12:13:48 pm] | Sysprep-UEFI-x64: No need to sync d1p1.img file to Ashford [07-07-18 12:13:47 pm] | Sysprep-UEFI-x64: No need to sync d1.partitions file to Ashford [07-07-18 12:13:47 pm] | Sysprep-UEFI-x64: No need to sync d1.original.uuids file to Ashford [07-07-18 12:13:46 pm] | Sysprep-UEFI-x64: No need to sync d1.original.swapuuids file to Ashford [07-07-18 12:13:46 pm] | Sysprep-UEFI-x64: No need to sync d1.original.fstypes file to Ashford [07-07-18 12:13:45 pm] | Sysprep-UEFI-x64: No need to sync d1.minimum.partitions file to Ashford [07-07-18 12:13:45 pm] | Sysprep-UEFI-x64: No need to sync d1.mbr file to Ashford [07-07-18 12:13:44 pm] | Sysprep-UEFI-x64: No need to sync d1.fixed_size_partitions file to Ashford [07-07-18 12:13:42 pm] | Image Name: Sysprep-UEFI-x64 [07-07-18 12:13:42 pm] * Found Image to transfer to 9 s [07-07-18 12:13:39 pm] | There are no other members to sync to. [07-07-18 12:13:39 pm] | Image Name: Sysprep-All (Win 7) [07-07-18 12:13:39 pm] * Not syncing Image between s [07-07-18 12:13:39 pm] | There are no other members to sync to. [07-07-18 12:13:39 pm] | Image Name: [07-07-18 12:13:39 pm] * Not syncing Image between s [07-07-18 12:13:38 pm] | Replicating postinitscripts [07-07-18 12:13:38 pm] | There are no other members to sync to. [07-07-18 12:13:38 pm] | Image Name: [07-07-18 12:13:38 pm] * Not syncing Image between s [07-07-18 12:13:38 pm] | Replicating postdownloadscripts [07-07-18 12:13:38 pm] * Attempting to perform Group -> Group image replication. [07-07-18 12:13:38 pm] * We are node ID: 1. We are node name: 1Fog-Server [07-07-18 12:13:38 pm] * We are group ID: 1. We are group name: 1Office [07-07-18 12:13:38 pm] * Starting Image Replication.
FOG 1.5.4.8
Furthermore…after disabling the Ashford node it does in fact go to the second “Ramage” node, but stops there and doesn’t try to replicate to the other nodes whatsoever…(I have 9 nodes alltogether, including the main server)
[07-07-18 1:11:30 pm] | There are no other members to sync to. [07-07-18 1:11:30 pm] | Image Name: SysprepWin10EDU [07-07-18 1:11:30 pm] * Not syncing Image between s [07-07-18 1:11:30 pm] | There are no other members to sync to. [07-07-18 1:11:30 pm] | Image Name: Sysprep-Win10EDU-X64 [07-07-18 1:11:29 pm] * Not syncing Image between s [07-07-18 1:11:29 pm] | There are no other members to sync to. [07-07-18 1:11:29 pm] | Image Name: Sysprep-UEFI-x64 [07-07-18 1:11:29 pm] * Not syncing Image between s [07-07-18 1:11:29 pm] | There are no other members to sync to. [07-07-18 1:11:28 pm] | Image Name: Sysprep-All (Win 7) [07-07-18 1:11:28 pm] * Not syncing Image between s [07-07-18 1:11:28 pm] * Attempting to perform Group -> Nodes image replication. [07-07-18 1:11:28 pm] | Replication already running with PID: 55853 [07-07-18 1:11:28 pm] | Replication already running with PID: 55853 [07-07-18 1:11:28 pm] | Replication already running with PID: 55853 [07-07-18 1:11:28 pm] | Replication already running with PID: 55853 [07-07-18 1:11:28 pm] | Replication already running with PID: 55853 [07-07-18 1:11:28 pm] | Replication already running with PID: 55853 [07-07-18 1:11:28 pm] * Started sync for Image SysprepWin10EDU lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.SysprepWin10EDU.transfer.Ramage.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepWin10EDU" "/images/SysprepWin10EDU"; exit' -u fog,[Protected] 10.72.216.50 [07-07-18 1:11:28 pm] | CMD: [07-07-18 1:11:28 pm] * Starting Sync Actions [07-07-18 1:11:28 pm] | d1p1.img File does not exist. Ramage [07-07-18 1:11:28 pm] | Files do not match on server: Ramage [07-07-18 1:11:20 pm] | d1.partitions File does not exist. Ramage [07-07-18 1:11:20 pm] | Files do not match on server: Ramage [07-07-18 1:11:13 pm] | d1.original.swapuuids File does not exist. Ramage [07-07-18 1:11:13 pm] | Files do not match on server: Ramage [07-07-18 1:11:09 pm] | d1.original.fstypes File does not exist. Ramage [07-07-18 1:11:09 pm] | Files do not match on server: Ramage [07-07-18 1:11:05 pm] | d1.minimum.partitions File does not exist. Ramage [07-07-18 1:11:05 pm] | Files do not match on server: Ramage [07-07-18 1:10:59 pm] | d1.mbr File does not exist. Ramage [07-07-18 1:10:59 pm] | Files do not match on server: Ramage [07-07-18 1:10:51 pm] | d1.fixed_size_partitions File does not exist. Ramage [07-07-18 1:10:51 pm] | Files do not match on server: Ramage [07-07-18 1:10:40 pm] | Image Name: SysprepWin10EDU [07-07-18 1:10:40 pm] * Found Image to transfer to 8 s [07-07-18 1:10:33 pm] | Replication already running with PID: 51510 [07-07-18 1:10:33 pm] | Replication already running with PID: 51510 [07-07-18 1:10:33 pm] | Replication already running with PID: 51510 [07-07-18 1:10:33 pm] | Replication already running with PID: 51510 [07-07-18 1:10:33 pm] | Replication already running with PID: 51510 [07-07-18 1:10:33 pm] | Replication already running with PID: 51510 [07-07-18 1:10:33 pm] * Started sync for Image Sysprep-Win10EDU-X64 lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-Win10EDU-X64.transfer.Ramage.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepImagex64" "/images/SysprepImagex64"; exit' -u fog,[Protected] 10.72.216.50 [07-07-18 1:10:33 pm] | CMD: [07-07-18 1:10:33 pm] * Starting Sync Actions [07-07-18 1:10:33 pm] | d1p2.img File does not exist. Ramage [07-07-18 1:10:33 pm] | Files do not match on server: Ramage [07-07-18 1:10:28 pm] | d1p1.img File does not exist. Ramage [07-07-18 1:10:28 pm] | Files do not match on server: Ramage [07-07-18 1:10:24 pm] | d1.partitions File does not exist. Ramage [07-07-18 1:10:24 pm] | Files do not match on server: Ramage [07-07-18 1:10:20 pm] | d1.original.swapuuids File does not exist. Ramage [07-07-18 1:10:20 pm] | Files do not match on server: Ramage [07-07-18 1:10:14 pm] | d1.original.fstypes File does not exist. Ramage [07-07-18 1:10:14 pm] | Files do not match on server: Ramage [07-07-18 1:10:11 pm] | d1.minimum.partitions File does not exist. Ramage [07-07-18 1:10:11 pm] | Files do not match on server: Ramage [07-07-18 1:10:09 pm] | d1.mbr File does not exist. Ramage [07-07-18 1:10:09 pm] | Files do not match on server: Ramage [07-07-18 1:10:07 pm] | d1.fixed_size_partitions File does not exist. Ramage [07-07-18 1:10:07 pm] | Files do not match on server: Ramage [07-07-18 1:10:01 pm] | Image Name: Sysprep-Win10EDU-X64 [07-07-18 1:10:01 pm] * Found Image to transfer to 8 s [07-07-18 1:09:57 pm] | Replication already running with PID: 48074 [07-07-18 1:09:57 pm] | Replication already running with PID: 48074 [07-07-18 1:09:57 pm] | Replication already running with PID: 48074 [07-07-18 1:09:57 pm] | Replication already running with PID: 48074 [07-07-18 1:09:57 pm] | Replication already running with PID: 48074 [07-07-18 1:09:57 pm] | Replication already running with PID: 48074 [07-07-18 1:09:57 pm] * Started sync for Image Sysprep-UEFI-x64 lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-UEFI-x64.transfer.Ramage.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Sysprep-UEFI-x64" "/images/Sysprep-UEFI-x64"; exit' -u fog,[Protected] 10.72.216.50 [07-07-18 1:09:57 pm] | CMD: [07-07-18 1:09:57 pm] * Starting Sync Actions [07-07-18 1:09:57 pm] | d1p4.img File does not exist. Ramage [07-07-18 1:09:57 pm] | Files do not match on server: Ramage [07-07-18 1:09:56 pm] | d1p3.img File does not exist. Ramage [07-07-18 1:09:56 pm] | Files do not match on server: Ramage [07-07-18 1:09:56 pm] | d1p2.img File does not exist. Ramage [07-07-18 1:09:56 pm] | Files do not match on server: Ramage [07-07-18 1:09:55 pm] | d1p1.img File does not exist. Ramage [07-07-18 1:09:55 pm] | Files do not match on server: Ramage [07-07-18 1:09:55 pm] | d1.partitions File does not exist. Ramage [07-07-18 1:09:55 pm] | Files do not match on server: Ramage [07-07-18 1:09:54 pm] | d1.original.uuids File does not exist. Ramage [07-07-18 1:09:54 pm] | Files do not match on server: Ramage [07-07-18 1:09:54 pm] | d1.original.swapuuids File does not exist. Ramage [07-07-18 1:09:54 pm] | Files do not match on server: Ramage [07-07-18 1:09:53 pm] | d1.original.fstypes File does not exist. Ramage [07-07-18 1:09:53 pm] | Files do not match on server: Ramage [07-07-18 1:09:53 pm] | d1.minimum.partitions File does not exist. Ramage [07-07-18 1:09:53 pm] | Files do not match on server: Ramage [07-07-18 1:09:52 pm] | d1.mbr File does not exist. Ramage [07-07-18 1:09:52 pm] | Files do not match on server: Ramage [07-07-18 1:09:52 pm] | d1.fixed_size_partitions File does not exist. Ramage [07-07-18 1:09:52 pm] | Files do not match on server: Ramage [07-07-18 1:09:50 pm] | Image Name: Sysprep-UEFI-x64 [07-07-18 1:09:50 pm] * Found Image to transfer to 8 s [07-07-18 1:09:48 pm] | There are no other members to sync to. [07-07-18 1:09:48 pm] | Image Name: Sysprep-All (Win 7) [07-07-18 1:09:48 pm] * Not syncing Image between s [07-07-18 1:09:47 pm] | There are no other members to sync to. [07-07-18 1:09:47 pm] | Image Name: [07-07-18 1:09:47 pm] * Not syncing Image between s [07-07-18 1:09:47 pm] | Replicating postinitscripts [07-07-18 1:09:47 pm] | There are no other members to sync to. [07-07-18 1:09:47 pm] | Image Name: [07-07-18 1:09:47 pm] * Not syncing Image between s [07-07-18 1:09:46 pm] | Replicating postdownloadscripts [07-07-18 1:09:46 pm] * Attempting to perform Group -> Group image replication. [07-07-18 1:09:46 pm] * We are node ID: 1. We are node name: 1Fog-Server [07-07-18 1:09:46 pm] * We are group ID: 1. We are group name: 1Office [07-07-18 1:09:46 pm] * Starting Image Replication. [07-07-18 1:09:43 pm] * Starting service loop [07-07-18 1:09:43 pm] * Checking for new items every 4800 seconds [07-07-18 1:09:43 pm] * Starting ImageReplicator Service [07-07-18 1:09:43 pm] Interface Ready with IP Address: fog-server.boone.edu [07-07-18 1:09:43 pm] Interface Ready with IP Address: 127.0.1.1 [07-07-18 1:09:43 pm] Interface Ready with IP Address: 127.0.0.1 [07-07-18 1:09:43 pm] Interface Ready with IP Address: 10.72.3.50 ================================== == Released under GPL Version 3 == ================================== = https://fogproject.org/Credits = ============ Credits ============= ================================== ===== Free Opensource Ghost ====== ================================== === ========== ===== ==== === ========= == === == === === ======== ==== == ==== === === ======== ==== == === === === ==== ==== == ========= === ======== ==== == ========= === ======== ==== == ==== === === ========= == === == === === ==== ===== ==== ================================== [07-07-18 1:09:43 pm]
After disabling each node in order after replication, I can get images replicated to all nodes.
-
@hanz et al.
Mind updating again to latest working, I didn’t push a version change yet, I just pushed a quick fix to what I hope will help out.
Essentially, I’m setting the variable for checking running processes. However, the variable was accidentally unset within its running scope, so I re-added that variable so it should be used. Maybe this is why the weirdness was happening? It was always checking the running process of the first item in the list, so while it did the checks on the other nodes, only the first node was being checked for a running process (hence why disabling that first node would allow the next, and so forth, to start replicating properly.)
It’s a long shot, and I don’t have a means to test myself. Please just run a
git pull
and see if that helps out at all. -
@tom-elliott You are the man sir…The replication does indeed appear to be working as expected…I removed files from one node to confirm complete transfer. The only question I have concerns the odd output in Image Replicator log showing the following:
[07-09-18 9:22:55 am] * Started sync for Image Sysprep-Win10EDU-X64 lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-Win10EDU-X64.transfer.VanHigh.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepImagex64" "/images/SysprepImagex64"; exit' -u fog,[Protected] 10.73.176.50 [07-09-18 9:22:55 am] | CMD: [07-09-18 9:22:55 am] | Deleting remote file: /images/SysprepImagex64/d1p2.img [07-09-18 9:22:55 am] | Files do not match on server: VanHigh [07-09-18 9:21:02 am] | Sysprep-Win10EDU-X64: No need to sync d1p1.img file to VanHigh [07-09-18 9:21:01 am] | Sysprep-Win10EDU-X64: No need to sync d1.partitions file to VanHigh [07-09-18 9:21:01 am] | Sysprep-Win10EDU-X64: No need to sync d1.original.swapuuids file to VanHigh [07-09-18 9:21:01 am] | Sysprep-Win10EDU-X64: No need to sync d1.original.fstypes file to VanHigh [07-09-18 9:21:00 am] | Sysprep-Win10EDU-X64: No need to sync d1.minimum.partitions file to VanHigh [07-09-18 9:21:00 am] | Sysprep-Win10EDU-X64: No need to sync d1.mbr file to VanHigh [07-09-18 9:20:59 am] | Sysprep-Win10EDU-X64: No need to sync d1.fixed_size_partitions file to VanHigh [07-09-18 9:20:58 am] * Started sync for Image Sysprep-Win10EDU-X64 lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Sysprep-Win10EDU-X64.transfer.VanElem.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; set net:limit-total-rate 0:128000000; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/SysprepImagex64" "/images/SysprepImagex64"; exit' -u fog,[Protected] 10.72.200.50 [07-09-18 9:20:58 am] | CMD: [07-09-18 9:20:58 am] | Deleting remote file: /images/SysprepImagex64/d1p2.img [07-09-18 9:20:58 am] | Files do not match on server: VanElem [07-09-18 9:18:57 am] | Sysprep-Win10EDU-X64: No need to sync d1p1.img file to VanElem [07-09-18 9:18:55 am] | Sysprep-Win10EDU-X64: No need to sync d1.partitions file to VanElem [07-09-18 9:18:54 am] | Sysprep-Win10EDU-X64: No need to sync d1.original.swapuuids file to VanElem [07-09-18 9:18:53 am] | Sysprep-Win10EDU-X64: No need to sync d1.original.fstypes file to VanElem [07-09-18 9:18:53 am] | Sysprep-Win10EDU-X64: No need to sync d1.minimum.partitions file to VanElem [07-09-18 9:18:52 am] | Sysprep-Win10EDU-X64: No need to sync d1.mbr file to VanElem [07-09-18 9:18:51 am] | Sysprep-Win10EDU-X64: No need to sync d1.fixed_size_partitions file to VanElem
The files are there already and not actually being deleted and resent as the log indicates…I only attached the logs from 2 nodes, but they all behave the same and have the same output when image is already present. This does seem to only affect the largest partition.
-
@hanz @Tom-Elliott I updated this morning and can confirm that my server will now check other nodes after the first one in line has files to replace. However, I believe it is still having the issue of the files saying they are different even though the md5sum is the same (and not deleting said file when it says it will and transfer to log)
[07-09-18 7:04:19 am] ================================== === ==== ===== ==== === ========= == === == === === ======== ==== == ==== === === ======== ==== == ========= === ==== ==== == ========= === ======== ==== == === === === ======== ==== == ==== === === ========= == === == === === ========== ===== ==== ================================== ===== Free Opensource Ghost ====== ================================== ============ Credits ============= = https://fogproject.org/Credits = ================================== == Released under GPL Version 3 == ================================== [07-09-18 7:04:19 am] Interface Ready with IP Address: 10.59.10.12 [07-09-18 7:04:19 am] Interface Ready with IP Address: 127.0.0.1 [07-09-18 7:04:19 am] Interface Ready with IP Address: 127.0.1.1 [07-09-18 7:04:19 am] Interface Ready with IP Address: 192.168.122.1 [07-09-18 7:04:19 am] Interface Ready with IP Address: do-isdir-01.tsd.tsdnet.turlock.k12.ca.us [07-09-18 7:04:19 am] Interface Ready with IP Address: fog-master-tsd-tsdnet-turlock-k12-ca-us [07-09-18 7:04:19 am] * Starting ImageReplicator Service [07-09-18 7:04:19 am] * Checking for new items every 900 seconds [07-09-18 7:04:19 am] * Starting service loop [07-09-18 7:04:19 am] * Starting Image Replication. [07-09-18 7:04:19 am] * We are group ID: 1. We are group name: default [07-09-18 7:04:19 am] * We are node ID: 1. We are node name: DO-FOG-MASTER [07-09-18 7:04:19 am] * Attempting to perform Group -> Group image replication. [07-09-18 7:04:19 am] | Replicating postdownloadscripts [07-09-18 7:04:20 am] * Found Image to transfer to 2 s [07-09-18 7:04:20 am] | File Name: postdownloadscripts [07-09-18 7:04:29 am] | postdownloadscripts: No need to sync fog.deletelog file to OC-FOG-NODE [07-09-18 7:04:29 am] | postdownloadscripts: No need to sync fog.drivers file to OC-FOG-NODE [07-09-18 7:04:29 am] | postdownloadscripts: No need to sync fog.postdownload file to OC-FOG-NODE [07-09-18 7:04:29 am] * All files synced for this item. [07-09-18 7:04:48 am] | postdownloadscripts: No need to sync fog.deletelog file to PH-FOG-NODE [07-09-18 7:04:48 am] | Files do not match on server: PH-FOG-NODE [07-09-18 7:04:48 am] | Deleting remote file: /images/postdownloadscripts/fog.drivers [07-09-18 7:04:48 am] | Files do not match on server: PH-FOG-NODE [07-09-18 7:04:48 am] | Deleting remote file: /images/postdownloadscripts/fog.postdownload [07-09-18 7:04:48 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.PH-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 10.60.219.12 [07-09-18 7:04:48 am] * Started sync for Image postdownloadscripts [07-09-18 7:04:48 am] | Replicating postinitscripts [07-09-18 7:04:48 am] * Found Image to transfer to 2 s [07-09-18 7:04:48 am] | File Name: dev/postinitscripts [07-09-18 7:04:57 am] | dev/postinitscripts: No need to sync fog.postinit file to OC-FOG-NODE [07-09-18 7:04:57 am] * All files synced for this item. [07-09-18 7:05:06 am] | dev/postinitscripts: No need to sync fog.postinit file to PH-FOG-NODE [07-09-18 7:05:06 am] * All files synced for this item. [07-09-18 7:05:06 am] * Not syncing Image between s [07-09-18 7:05:06 am] | Image Name: Win10BaseVM [07-09-18 7:05:06 am] | There are no other members to sync to. [07-09-18 7:05:06 am] * Not syncing Image between s [07-09-18 7:05:06 am] | Image Name: Win7BaseVM [07-09-18 7:05:06 am] | There are no other members to sync to. [07-09-18 7:05:06 am] * Attempting to perform Group -> Nodes image replication. [07-09-18 7:05:07 am] * Found Image to transfer to 2 s [07-09-18 7:05:07 am] | Image Name: Win10BaseVM [07-09-18 7:05:16 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [07-09-18 7:05:17 am] | Win10BaseVM: No need to sync d1.mbr file to OC-FOG-NODE [07-09-18 7:05:17 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [07-09-18 7:05:17 am] | Win10BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE [07-09-18 7:05:17 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [07-09-18 7:05:18 am] | Win10BaseVM: No need to sync d1.original.uuids file to OC-FOG-NODE [07-09-18 7:05:18 am] | Win10BaseVM: No need to sync d1.partitions file to OC-FOG-NODE [07-09-18 7:05:18 am] | Win10BaseVM: No need to sync d1p1.img file to OC-FOG-NODE [07-09-18 7:05:19 am] | Win10BaseVM: No need to sync d1p2.img file to OC-FOG-NODE [07-09-18 7:05:19 am] | Win10BaseVM: No need to sync d1p3.img file to OC-FOG-NODE [07-09-18 7:07:17 am] | Files do not match on server: OC-FOG-NODE [07-09-18 7:07:17 am] | Deleting remote file: /images/Win10BaseVM/d1p4.img [07-09-18 7:07:17 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win10BaseVM.transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win10BaseVM" "/images/Win10BaseVM"; exit' -u fog,[Protected] 10.59.181.12 [07-09-18 7:07:17 am] * Started sync for Image Win10BaseVM [07-09-18 7:07:26 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE [07-09-18 7:07:26 am] | Win10BaseVM: No need to sync d1.mbr file to PH-FOG-NODE [07-09-18 7:07:27 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to PH-FOG-NODE [07-09-18 7:07:27 am] | Win10BaseVM: No need to sync d1.original.fstypes file to PH-FOG-NODE [07-09-18 7:07:27 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [07-09-18 7:07:27 am] | Win10BaseVM: No need to sync d1.original.uuids file to PH-FOG-NODE [07-09-18 7:07:28 am] | Win10BaseVM: No need to sync d1.partitions file to PH-FOG-NODE [07-09-18 7:07:28 am] | Win10BaseVM: No need to sync d1p1.img file to PH-FOG-NODE [07-09-18 7:07:28 am] | Win10BaseVM: No need to sync d1p2.img file to PH-FOG-NODE [07-09-18 7:07:29 am] | Win10BaseVM: No need to sync d1p3.img file to PH-FOG-NODE [07-09-18 7:09:28 am] | Win10BaseVM: No need to sync d1p4.img file to PH-FOG-NODE [07-09-18 7:09:28 am] * All files synced for this item. [07-09-18 7:09:28 am] * Found Image to transfer to 2 s [07-09-18 7:09:28 am] | Image Name: Win7BaseVM [07-09-18 7:09:37 am] | Win7BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [07-09-18 7:09:37 am] | Win7BaseVM: No need to sync d1.mbr file to OC-FOG-NODE [07-09-18 7:09:37 am] | Win7BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [07-09-18 7:09:38 am] | Win7BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE [07-09-18 7:09:38 am] | Win7BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [07-09-18 7:09:38 am] | Win7BaseVM: No need to sync d1.partitions file to OC-FOG-NODE [07-09-18 7:11:56 am] | Win7BaseVM: No need to sync d1p1.img file to OC-FOG-NODE [07-09-18 7:11:56 am] * All files synced for this item. [07-09-18 7:12:05 am] | Win7BaseVM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE [07-09-18 7:12:05 am] | Win7BaseVM: No need to sync d1.mbr file to PH-FOG-NODE [07-09-18 7:12:05 am] | Win7BaseVM: No need to sync d1.minimum.partitions file to PH-FOG-NODE [07-09-18 7:12:06 am] | Win7BaseVM: No need to sync d1.original.fstypes file to PH-FOG-NODE [07-09-18 7:12:06 am] | Win7BaseVM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [07-09-18 7:12:06 am] | Win7BaseVM: No need to sync d1.partitions file to PH-FOG-NODE [07-09-18 7:14:21 am] | Win7BaseVM: No need to sync d1p1.img file to PH-FOG-NODE [07-09-18 7:14:21 am] * All files synced for this item. [07-09-18 7:29:21 am] * Starting Image Replication. [07-09-18 7:29:21 am] * We are group ID: 1. We are group name: default [07-09-18 7:29:21 am] * We are node ID: 1. We are node name: DO-FOG-MASTER [07-09-18 7:29:21 am] * Attempting to perform Group -> Group image replication. [07-09-18 7:29:21 am] | Replicating postdownloadscripts [07-09-18 7:29:21 am] * Found Image to transfer to 2 s [07-09-18 7:29:21 am] | File Name: postdownloadscripts [07-09-18 7:29:30 am] | postdownloadscripts: No need to sync fog.deletelog file to OC-FOG-NODE [07-09-18 7:29:31 am] | postdownloadscripts: No need to sync fog.drivers file to OC-FOG-NODE [07-09-18 7:29:31 am] | postdownloadscripts: No need to sync fog.postdownload file to OC-FOG-NODE [07-09-18 7:29:31 am] * All files synced for this item. [07-09-18 7:29:40 am] | postdownloadscripts: No need to sync fog.deletelog file to PH-FOG-NODE [07-09-18 7:29:41 am] | postdownloadscripts: No need to sync fog.drivers file to PH-FOG-NODE [07-09-18 7:29:41 am] | postdownloadscripts: No need to sync fog.postdownload file to PH-FOG-NODE [07-09-18 7:29:41 am] * All files synced for this item. [07-09-18 7:29:41 am] | Replicating postinitscripts [07-09-18 7:29:41 am] * Found Image to transfer to 2 s [07-09-18 7:29:41 am] | File Name: dev/postinitscripts [07-09-18 7:29:51 am] | dev/postinitscripts: No need to sync fog.postinit file to OC-FOG-NODE [07-09-18 7:29:51 am] * All files synced for this item. [07-09-18 7:30:00 am] | dev/postinitscripts: No need to sync fog.postinit file to PH-FOG-NODE [07-09-18 7:30:00 am] * All files synced for this item. [07-09-18 7:30:00 am] * Not syncing Image between s [07-09-18 7:30:00 am] | Image Name: Win10BaseVM [07-09-18 7:30:00 am] | There are no other members to sync to. [07-09-18 7:30:00 am] * Not syncing Image between s [07-09-18 7:30:00 am] | Image Name: Win7BaseVM [07-09-18 7:30:00 am] | There are no other members to sync to. [07-09-18 7:30:00 am] * Attempting to perform Group -> Nodes image replication. [07-09-18 7:30:00 am] * Found Image to transfer to 2 s [07-09-18 7:30:00 am] | Image Name: Win10BaseVM [07-09-18 7:30:10 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to OC-FOG-NODE [07-09-18 7:30:10 am] | Win10BaseVM: No need to sync d1.mbr file to OC-FOG-NODE [07-09-18 7:30:10 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to OC-FOG-NODE [07-09-18 7:30:11 am] | Win10BaseVM: No need to sync d1.original.fstypes file to OC-FOG-NODE [07-09-18 7:30:11 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to OC-FOG-NODE [07-09-18 7:30:11 am] | Win10BaseVM: No need to sync d1.original.uuids file to OC-FOG-NODE [07-09-18 7:30:11 am] | Win10BaseVM: No need to sync d1.partitions file to OC-FOG-NODE [07-09-18 7:30:12 am] | Win10BaseVM: No need to sync d1p1.img file to OC-FOG-NODE [07-09-18 7:30:12 am] | Win10BaseVM: No need to sync d1p2.img file to OC-FOG-NODE [07-09-18 7:30:13 am] | Win10BaseVM: No need to sync d1p3.img file to OC-FOG-NODE [07-09-18 7:32:18 am] | Files do not match on server: OC-FOG-NODE [07-09-18 7:32:18 am] | Deleting remote file: /images/Win10BaseVM/d1p4.img [07-09-18 7:32:18 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.Win10BaseVM.transfer.OC-FOG-NODE.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/Win10BaseVM" "/images/Win10BaseVM"; exit' -u fog,[Protected] 10.59.181.12 [07-09-18 7:32:18 am] * Started sync for Image Win10BaseVM [07-09-18 7:32:28 am] | Win10BaseVM: No need to sync d1.fixed_size_partitions file to PH-FOG-NODE [07-09-18 7:32:28 am] | Win10BaseVM: No need to sync d1.mbr file to PH-FOG-NODE [07-09-18 7:32:28 am] | Win10BaseVM: No need to sync d1.minimum.partitions file to PH-FOG-NODE [07-09-18 7:32:29 am] | Win10BaseVM: No need to sync d1.original.fstypes file to PH-FOG-NODE [07-09-18 7:32:29 am] | Win10BaseVM: No need to sync d1.original.swapuuids file to PH-FOG-NODE [07-09-18 7:32:29 am] | Win10BaseVM: No need to sync d1.original.uuids file to PH-FOG-NODE [07-09-18 7:32:29 am] | Win10BaseVM: No need to sync d1.partitions file to PH-FOG-NODE [07-09-18 7:32:30 am] | Win10BaseVM: No need to sync d1p1.img file to PH-FOG-NODE [07-09-18 7:32:30 am] | Win10BaseVM: No need to sync d1p2.img file to PH-FOG-NODE [07-09-18 7:32:31 am] | Win10BaseVM: No need to sync d1p3.img file to PH-FOG-NODE```
-
@jflippen I to updated and noticed it is no longer in a replication loop. Interesting as I observed the replication log go through all my storage nodes is that first storage node has same file that does not match and one other storage node. So i’m waiting for next round of replication to occur and continue to observe. Update seem to fix the processes that @Tom-Elliott was talking about earlier and seems to check other storage nodes as replication goes down the line of storage nodes.
-
I got a undefined variable error in replication log at the bottom. As you can see, replication is working but still states that d1p2.img file doesn’t match. Also, the file for the storage node with the error was perfectly fine on an earlier check. I didn’t update any images today.
[07-09-18 5:51:14 am] | There are no other members to sync to. [07-09-18 5:51:17 am] * Found Image to transfer to 8 s [07-09-18 5:51:17 am] | Image Name: 32-Dell-790 [07-09-18 5:51:18 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to BCS-Master [07-09-18 5:51:18 am] | 32-Dell-790: No need to sync d1.mbr file to BCS-Master [07-09-18 5:51:19 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to BCS-Master [07-09-18 5:51:20 am] | 32-Dell-790: No need to sync d1.original.fstypes file to BCS-Master [07-09-18 5:51:21 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to BCS-Master [07-09-18 5:51:21 am] | 32-Dell-790: No need to sync d1.partitions file to BCS-Master [07-09-18 5:51:23 am] | 32-Dell-790: No need to sync d1p1.img file to BCS-Master [07-09-18 5:58:23 am] | 32-Dell-790: No need to sync d1p2.img file to BCS-Master [07-09-18 5:58:23 am] * All files synced for this item. [07-09-18 5:58:24 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to CHS-Master [07-09-18 5:58:24 am] | 32-Dell-790: No need to sync d1.mbr file to CHS-Master [07-09-18 5:58:25 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to CHS-Master [07-09-18 5:58:25 am] | 32-Dell-790: No need to sync d1.original.fstypes file to CHS-Master [07-09-18 5:58:26 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to CHS-Master [07-09-18 5:58:26 am] | 32-Dell-790: No need to sync d1.partitions file to CHS-Master [07-09-18 5:58:26 am] | 32-Dell-790: No need to sync d1p1.img file to CHS-Master [07-09-18 6:03:39 am] | 32-Dell-790: No need to sync d1p2.img file to CHS-Master [07-09-18 6:03:39 am] * All files synced for this item. [07-09-18 6:03:39 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to ESC-Master [07-09-18 6:03:40 am] | 32-Dell-790: No need to sync d1.mbr file to ESC-Master [07-09-18 6:03:40 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to ESC-Master [07-09-18 6:03:40 am] | 32-Dell-790: No need to sync d1.original.fstypes file to ESC-Master [07-09-18 6:03:41 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to ESC-Master [07-09-18 6:03:41 am] | 32-Dell-790: No need to sync d1.partitions file to ESC-Master [07-09-18 6:03:41 am] | 32-Dell-790: No need to sync d1p1.img file to ESC-Master [07-09-18 6:07:44 am] | 32-Dell-790: No need to sync d1p2.img file to ESC-Master [07-09-18 6:07:44 am] * All files synced for this item. [07-09-18 6:07:45 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to Jep-Master [07-09-18 6:07:45 am] | 32-Dell-790: No need to sync d1.mbr file to Jep-Master [07-09-18 6:07:46 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to Jep-Master [07-09-18 6:07:46 am] | 32-Dell-790: No need to sync d1.original.fstypes file to Jep-Master [07-09-18 6:07:47 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to Jep-Master [07-09-18 6:07:47 am] | 32-Dell-790: No need to sync d1.partitions file to Jep-Master [07-09-18 6:07:48 am] | 32-Dell-790: No need to sync d1p1.img file to Jep-Master [07-09-18 6:13:05 am] | 32-Dell-790: No need to sync d1p2.img file to Jep-Master [07-09-18 6:13:05 am] * All files synced for this item. [07-09-18 6:13:06 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VHS-Master [07-09-18 6:13:07 am] | 32-Dell-790: No need to sync d1.mbr file to VHS-Master [07-09-18 6:13:07 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to VHS-Master [07-09-18 6:13:08 am] | 32-Dell-790: No need to sync d1.original.fstypes file to VHS-Master [07-09-18 6:13:08 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to VHS-Master [07-09-18 6:13:09 am] | 32-Dell-790: No need to sync d1.partitions file to VHS-Master [07-09-18 6:13:10 am] | 32-Dell-790: No need to sync d1p1.img file to VHS-Master [07-09-18 6:19:08 am] | 32-Dell-790: No need to sync d1p2.img file to VHS-Master [07-09-18 6:19:08 am] * All files synced for this item. [07-09-18 6:19:10 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VP-Master [07-09-18 6:19:11 am] | 32-Dell-790: No need to sync d1.mbr file to VP-Master [07-09-18 6:19:11 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to VP-Master [07-09-18 6:19:12 am] | 32-Dell-790: No need to sync d1.original.fstypes file to VP-Master [07-09-18 6:19:13 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to VP-Master [07-09-18 6:19:13 am] | 32-Dell-790: No need to sync d1.partitions file to VP-Master [07-09-18 6:19:14 am] | 32-Dell-790: No need to sync d1p1.img file to VP-Master [07-09-18 6:28:00 am] | 32-Dell-790: No need to sync d1p2.img file to VP-Master [07-09-18 6:28:00 am] * All files synced for this item. [07-09-18 6:28:01 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to WCW-Master [07-09-18 6:28:02 am] | 32-Dell-790: No need to sync d1.mbr file to WCW-Master [07-09-18 6:28:02 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to WCW-Master [07-09-18 6:28:03 am] | 32-Dell-790: No need to sync d1.original.fstypes file to WCW-Master [07-09-18 6:28:03 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to WCW-Master [07-09-18 6:28:04 am] | 32-Dell-790: No need to sync d1.partitions file to WCW-Master [07-09-18 6:28:05 am] | 32-Dell-790: No need to sync d1p1.img file to WCW-Master [07-09-18 6:34:02 am] | 32-Dell-790: No need to sync d1p2.img file to WCW-Master [07-09-18 6:34:02 am] * All files synced for this item. [07-09-18 6:34:04 am] * Found Image to transfer to 8 s [07-09-18 6:34:04 am] | Image Name: 64-Dell-790 [07-09-18 6:34:06 am] | 64-Dell-790: No need to sync d1.fixed_size_partitions file to BCS-Master [07-09-18 6:34:06 am] | 64-Dell-790: No need to sync d1.mbr file to BCS-Master [07-09-18 6:34:07 am] | 64-Dell-790: No need to sync d1.minimum.partitions file to BCS-Master [07-09-18 6:34:08 am] | 64-Dell-790: No need to sync d1.original.fstypes file to BCS-Master [07-09-18 6:34:08 am] | 64-Dell-790: No need to sync d1.original.swapuuids file to BCS-Master [07-09-18 6:34:09 am] | 64-Dell-790: No need to sync d1.partitions file to BCS-Master [07-09-18 6:34:10 am] | 64-Dell-790: No need to sync d1p1.img file to BCS-Master [07-09-18 6:41:33 am] | Files do not match on server: BCS-Master [07-09-18 6:41:33 am] | Deleting remote file: /images/64-Dell-790/d1p2.img [07-09-18 6:41:33 am] * Starting Sync Actions [07-09-18 6:41:33 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.64-Dell-790.transfer.BCS-Master.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/64-Dell-790" "/images/64-Dell-790"; exit' -u fog,[Protected] 10.210.100.61 [07-09-18 6:41:33 am] * Started sync for Image 64-Dell-790 [07-09-18 6:41:34 am] | 64-Dell-790: No need to sync d1.fixed_size_partitions file to CHS-Master [07-09-18 6:41:34 am] | 64-Dell-790: No need to sync d1.mbr file to CHS-Master [07-09-18 6:41:34 am] | 64-Dell-790: No need to sync d1.minimum.partitions file to CHS-Master [07-09-18 6:41:35 am] | 64-Dell-790: No need to sync d1.original.fstypes file to CHS-Master [07-09-18 6:41:35 am] | 64-Dell-790: No need to sync d1.original.swapuuids file to CHS-Master [07-09-18 6:41:36 am] | 64-Dell-790: No need to sync d1.partitions file to CHS-Master [07-09-18 6:41:36 am] | 64-Dell-790: No need to sync d1p1.img file to CHS-Master [07-09-18 6:47:24 am] | 64-Dell-790: No need to sync d1p2.img file to CHS-Master [07-09-18 6:47:24 am] * All files synced for this item. [07-09-18 6:47:25 am] | 64-Dell-790: No need to sync d1.fixed_size_partitions file to ESC-Master [07-09-18 6:47:25 am] | 64-Dell-790: No need to sync d1.mbr file to ESC-Master [07-09-18 6:47:25 am] | 64-Dell-790: No need to sync d1.minimum.partitions file to ESC-Master [07-09-18 6:47:25 am] | 64-Dell-790: No need to sync d1.original.fstypes file to ESC-Master [07-09-18 6:47:26 am] | 64-Dell-790: No need to sync d1.original.swapuuids file to ESC-Master [07-09-18 6:47:26 am] | 64-Dell-790: No need to sync d1.partitions file to ESC-Master [07-09-18 6:47:26 am] | 64-Dell-790: No need to sync d1p1.img file to ESC-Master [07-09-18 6:53:33 am] | 64-Dell-790: No need to sync d1p2.img file to ESC-Master [07-09-18 6:53:33 am] * All files synced for this item. [07-09-18 6:53:34 am] | 64-Dell-790: No need to sync d1.fixed_size_partitions file to Jep-Master [07-09-18 6:53:34 am] | 64-Dell-790: No need to sync d1.mbr file to Jep-Master [07-09-18 6:53:34 am] | 64-Dell-790: No need to sync d1.minimum.partitions file to Jep-Master [07-09-18 6:53:35 am] | 64-Dell-790: No need to sync d1.original.fstypes file to Jep-Master [07-09-18 6:53:35 am] | 64-Dell-790: No need to sync d1.original.swapuuids file to Jep-Master [07-09-18 6:53:36 am] | 64-Dell-790: No need to sync d1.partitions file to Jep-Master [07-09-18 6:53:36 am] | 64-Dell-790: No need to sync d1p1.img file to Jep-Master [07-09-18 6:59:23 am] | 64-Dell-790: No need to sync d1p2.img file to Jep-Master [07-09-18 6:59:23 am] * All files synced for this item. [07-09-18 6:59:24 am] | 64-Dell-790: No need to sync d1.fixed_size_partitions file to VHS-Master [07-09-18 6:59:24 am] | 64-Dell-790: No need to sync d1.mbr file to VHS-Master [07-09-18 6:59:25 am] | 64-Dell-790: No need to sync d1.minimum.partitions file to VHS-Master [07-09-18 6:59:25 am] | 64-Dell-790: No need to sync d1.original.fstypes file to VHS-Master [07-09-18 6:59:26 am] | 64-Dell-790: No need to sync d1.original.swapuuids file to VHS-Master [07-09-18 6:59:26 am] | 64-Dell-790: No need to sync d1.partitions file to VHS-Master [07-09-18 6:59:27 am] | 64-Dell-790: No need to sync d1p1.img file to VHS-Master [07-09-18 7:05:52 am] | 64-Dell-790: No need to sync d1p2.img file to VHS-Master [07-09-18 7:05:52 am] * All files synced for this item. [07-09-18 7:05:53 am] | 64-Dell-790: No need to sync d1.fixed_size_partitions file to VP-Master [07-09-18 7:05:54 am] | 64-Dell-790: No need to sync d1.mbr file to VP-Master [07-09-18 7:05:55 am] | 64-Dell-790: No need to sync d1.minimum.partitions file to VP-Master [07-09-18 7:05:55 am] | 64-Dell-790: No need to sync d1.original.fstypes file to VP-Master [07-09-18 7:05:56 am] | 64-Dell-790: No need to sync d1.original.swapuuids file to VP-Master [07-09-18 7:05:57 am] | 64-Dell-790: No need to sync d1.partitions file to VP-Master [07-09-18 7:05:58 am] | 64-Dell-790: No need to sync d1p1.img file to VP-Master [07-09-18 7:13:12 am] | Files do not match on server: VP-Master [07-09-18 7:13:12 am] | Deleting remote file: /images/64-Dell-790/d1p2.img [07-09-18 7:13:12 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.64-Dell-790.transfer.VP-Master.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/64-Dell-790" "/images/64-Dell-790"; exit' -u fog,[Protected] 10.205.100.61 [07-09-18 7:13:12 am] * Started sync for Image 64-Dell-790 [07-09-18 7:13:13 am] | 64-Dell-790: No need to sync d1.fixed_size_partitions file to WCW-Master [07-09-18 7:13:13 am] | 64-Dell-790: No need to sync d1.mbr file to WCW-Master [07-09-18 7:13:14 am] | 64-Dell-790: No need to sync d1.minimum.partitions file to WCW-Master [07-09-18 7:13:14 am] | 64-Dell-790: No need to sync d1.original.fstypes file to WCW-Master [07-09-18 7:13:15 am] | 64-Dell-790: No need to sync d1.original.swapuuids file to WCW-Master [07-09-18 7:13:15 am] | 64-Dell-790: No need to sync d1.partitions file to WCW-Master [07-09-18 7:13:16 am] | 64-Dell-790: No need to sync d1p1.img file to WCW-Master [07-09-18 7:19:33 am] | 64-Dell-790: No need to sync d1p2.img file to WCW-Master [07-09-18 7:19:33 am] * All files synced for this item. [07-09-18 7:19:33 am] * Attempting to perform Group -> Nodes image replication. [07-09-18 7:19:33 am] * Not syncing Image between s [07-09-18 7:19:33 am] | Image Name: 32-Dell-790 [07-09-18 7:19:33 am] | There are no other members to sync to. [07-09-18 7:19:34 am] * Not syncing Image between s [07-09-18 7:19:34 am] | Image Name: 64-Dell-790 [07-09-18 7:19:34 am] | There are no other members to sync to. [07-09-18 9:19:36 am] * Starting Image Replication. [07-09-18 9:19:36 am] * We are group ID: 1. We are group name: default [07-09-18 9:19:36 am] * We are node ID: 1. We are node name: DefaultMember [07-09-18 9:19:36 am] * Attempting to perform Group -> Group image replication. [07-09-18 9:19:36 am] | Replicating postdownloadscripts [07-09-18 9:19:37 am] * Not syncing Image between s [07-09-18 9:19:37 am] | Image Name: [07-09-18 9:19:37 am] | There are no other members to sync to. [07-09-18 9:19:37 am] | Replicating postinitscripts [07-09-18 9:19:37 am] * Not syncing Image between s [07-09-18 9:19:37 am] | Image Name: [07-09-18 9:19:37 am] | There are no other members to sync to. [07-09-18 9:19:40 am] * Found Image to transfer to 8 s [07-09-18 9:19:40 am] | Image Name: 32-Dell-790 [07-09-18 9:19:41 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to BCS-Master [07-09-18 9:19:42 am] | 32-Dell-790: No need to sync d1.mbr file to BCS-Master [07-09-18 9:19:43 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to BCS-Master [07-09-18 9:19:43 am] | 32-Dell-790: No need to sync d1.original.fstypes file to BCS-Master [07-09-18 9:19:44 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to BCS-Master [07-09-18 9:19:45 am] | 32-Dell-790: No need to sync d1.partitions file to BCS-Master [07-09-18 9:19:45 am] | 32-Dell-790: No need to sync d1p1.img file to BCS-Master [07-09-18 9:26:50 am] | 32-Dell-790: No need to sync d1p2.img file to BCS-Master [07-09-18 9:26:50 am] * All files synced for this item. [07-09-18 9:26:50 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to CHS-Master [07-09-18 9:26:51 am] | 32-Dell-790: No need to sync d1.mbr file to CHS-Master [07-09-18 9:26:51 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to CHS-Master [07-09-18 9:26:51 am] | 32-Dell-790: No need to sync d1.original.fstypes file to CHS-Master [07-09-18 9:26:52 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to CHS-Master [07-09-18 9:26:52 am] | 32-Dell-790: No need to sync d1.partitions file to CHS-Master [07-09-18 9:26:53 am] | 32-Dell-790: No need to sync d1p1.img file to CHS-Master [07-09-18 9:32:07 am] | 32-Dell-790: No need to sync d1p2.img file to CHS-Master [07-09-18 9:32:07 am] * All files synced for this item. [07-09-18 9:32:08 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to ESC-Master [07-09-18 9:32:08 am] | 32-Dell-790: No need to sync d1.mbr file to ESC-Master [07-09-18 9:32:08 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to ESC-Master [07-09-18 9:32:09 am] | 32-Dell-790: No need to sync d1.original.fstypes file to ESC-Master [07-09-18 9:32:09 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to ESC-Master [07-09-18 9:32:09 am] | 32-Dell-790: No need to sync d1.partitions file to ESC-Master [07-09-18 9:32:10 am] | 32-Dell-790: No need to sync d1p1.img file to ESC-Master [07-09-18 9:36:35 am] | 32-Dell-790: No need to sync d1p2.img file to ESC-Master [07-09-18 9:36:35 am] * All files synced for this item. [07-09-18 9:36:36 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to Jep-Master [07-09-18 9:36:36 am] | 32-Dell-790: No need to sync d1.mbr file to Jep-Master [07-09-18 9:36:37 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to Jep-Master [07-09-18 9:36:37 am] | 32-Dell-790: No need to sync d1.original.fstypes file to Jep-Master [07-09-18 9:36:37 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to Jep-Master [07-09-18 9:36:38 am] | 32-Dell-790: No need to sync d1.partitions file to Jep-Master [07-09-18 9:36:38 am] | 32-Dell-790: No need to sync d1p1.img file to Jep-Master [07-09-18 9:41:56 am] | 32-Dell-790: No need to sync d1p2.img file to Jep-Master [07-09-18 9:41:56 am] * All files synced for this item. [07-09-18 9:41:58 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VHS-Master [07-09-18 9:41:59 am] | 32-Dell-790: No need to sync d1.mbr file to VHS-Master [07-09-18 9:42:00 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to VHS-Master [07-09-18 9:42:00 am] | 32-Dell-790: No need to sync d1.original.fstypes file to VHS-Master [07-09-18 9:42:01 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to VHS-Master [07-09-18 9:42:01 am] | 32-Dell-790: No need to sync d1.partitions file to VHS-Master [07-09-18 9:42:02 am] | 32-Dell-790: No need to sync d1p1.img file to VHS-Master [07-09-18 9:49:02 am] | 32-Dell-790: No need to sync d1p2.img file to VHS-Master [07-09-18 9:49:02 am] * All files synced for this item. [07-09-18 9:49:03 am] | 32-Dell-790: No need to sync d1.fixed_size_partitions file to VP-Master [07-09-18 9:49:04 am] | 32-Dell-790: No need to sync d1.mbr file to VP-Master [07-09-18 9:49:05 am] | 32-Dell-790: No need to sync d1.minimum.partitions file to VP-Master [07-09-18 9:49:05 am] | 32-Dell-790: No need to sync d1.original.fstypes file to VP-Master [07-09-18 9:49:06 am] | 32-Dell-790: No need to sync d1.original.swapuuids file to VP-Master [07-09-18 9:49:07 am] | 32-Dell-790: No need to sync d1.partitions file to VP-Master [07-09-18 9:49:08 am] | 32-Dell-790: No need to sync d1p1.img file to VP-Master [07-09-18 9:56:38 am] | Files do not match on server: VP-Master [07-09-18 9:56:38 am] | Deleting remote file: /images/32-Dell-790/d1p2.img [07-09-18 9:56:38 am] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.32-Dell-790.transfer.VP-Master.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/32-Dell-790" "/images/32-Dell-790"; exit' -u fog,[Protected] 10.205.100.61 [07-09-18 9:56:38 am] * Started sync for Image 32-Dell-790 [07-09-18 9:56:38 am] * Type: 8, File: /var/www/fog/lib/service/fogservice.class.php, Line: 481, Message: Undefined variable: itemType, Host: 10.225.100.61, Username: fog