Replication Failure

Wayne Workman

r6483

Two storage nodes:

10.0.0.4 - Fedora 23 Server minimal, fully updated, r6483 - MASTER of the storage group.
10.0.0.5 - Fedora 23 Server minimal, fully updated, r6483 - not a master, but same storage group as the other node.

0_1456597474699_Storage Nodes.png

A while ago, I deployed one of my Fedora 23 images so that I could update it. After downloading and booting up - I ran dnf update. After a little while, it told me I had 600-some-odd-MB worth of updates that could be installed. I ran all of those updates, then I rebooted a few times, and then I re-captured my image once I was satisfied.

I sit and waited for replication to happen before I re-deployed to test my new image. I watched the replication logs, seeing that it checks about every 10 minutes.

I saw the bandwidth spike on my two storage nodes, transmitting from the master, receiving on the slave. I also saw notes in the logs about replication starting. I sit back and watched it, and waited for it to get done.

Once the bandwidth settled down on the nodes, the storage node usage pie graph showed that both are using the exact same amount of space:

0_1456597450171_Master storage node disk usage.png

0_1456597461068_Slave storage node disk usage.png

Checking the replication logs, I saw that there was some activity, but for a lot of stuff other than just my newly updated image:

[02-27-16 11:18:08 am] 
 ___ ___ ___ 
 /\ \ /\ \ /\ \ 
 /::\ \ /::\ \ /::\ \ 
 /:/\:\ \ /:/\:\ \ /:/\:\ \ 
 /::\-\:\ \ /:/ \:\ \ /:/ \:\ \ 
 /:/\:\ \:\__\ /:/__/ \:\__\ /:/__/_\:\__\ 
 \/__\:\ \/__/ \:\ \ /:/ / \:\ /\ \/__/ 
 \:\__\ \:\ /:/ / \:\ \:\__\ 
 \/__/ \:\/:/ / \:\/:/ / 
 \::/ / \::/ / 
 \/__/ \/__/ 

 ###########################################
 # Free Computer Imaging Solution #
 # Credits: #
 # http://fogproject.org/credits #
 # GNU GPL Version 3 #
 ###########################################

[02-27-16 11:18:09 am] Interface Ready with IP Address: 10.0.0.4
[02-27-16 11:18:09 am] Interface Ready with IP Address: 104.207.136.23
[02-27-16 11:18:09 am] Interface Ready with IP Address: fog-server
[02-27-16 11:18:09 am] * Starting ImageReplicator Service
[02-27-16 11:18:09 am] * Checking for new items every 600 seconds
[02-27-16 11:18:09 am] * Starting service loop
[02-27-16 11:18:10 am] * Starting Image Replication.
[02-27-16 11:18:10 am] * We are group ID: #1
[02-27-16 11:18:10 am] | We are group name: default
[02-27-16 11:18:10 am] * We have node ID: #1
[02-27-16 11:18:10 am] | We are node name: Master
[02-27-16 11:18:10 am] * Not syncing Image between group(s)
[02-27-16 11:18:10 am] | Image Name: CentOS7Optiplex745UpdateBASE
[02-27-16 11:18:10 am] | I am the only member
[02-27-16 11:18:10 am] * Not syncing Image between group(s)
[02-27-16 11:18:10 am] | Image Name: Fedora22LivingRoom
[02-27-16 11:18:10 am] | I am the only member
[02-27-16 11:18:10 am] * Not syncing Image between group(s)
[02-27-16 11:18:10 am] | Image Name: Fedora23LivingRoom
[02-27-16 11:18:10 am] | I am the only member
[02-27-16 11:18:10 am] * Not syncing Image between group(s)
[02-27-16 11:18:10 am] | Image Name: Win7
[02-27-16 11:18:10 am] | I am the only member
[02-27-16 11:18:10 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:18:10 am] | Image name: CentOS7Optiplex745UpdateBASE
[02-27-16 11:18:10 am] * Starting Sync Actions
[02-27-16 11:18:10 am] * Started sync for Image CentOS7Optiplex745UpdateBASE
[02-27-16 11:18:10 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:18:10 am] | Image name: Fedora22LivingRoom
[02-27-16 11:18:11 am] * Starting Sync Actions
[02-27-16 11:18:11 am] * Started sync for Image Fedora22LivingRoom
[02-27-16 11:18:11 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:18:11 am] | Image name: Fedora23LivingRoom
[02-27-16 11:18:11 am] * Starting Sync Actions
[02-27-16 11:18:11 am] * Started sync for Image Fedora23LivingRoom
[02-27-16 11:18:11 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:18:11 am] | Image name: Win7
[02-27-16 11:18:11 am] * Starting Sync Actions
[02-27-16 11:18:11 am] * Started sync for Image Win7
[02-27-16 11:28:23 am] * Starting Image Replication.
[02-27-16 11:28:23 am] * We are group ID: #1
[02-27-16 11:28:23 am] | We are group name: default
[02-27-16 11:28:23 am] * We have node ID: #1
[02-27-16 11:28:23 am] | We are node name: Master
[02-27-16 11:28:23 am] * Not syncing Image between group(s)
[02-27-16 11:28:23 am] | Image Name: CentOS7Optiplex745UpdateBASE
[02-27-16 11:28:23 am] | I am the only member
[02-27-16 11:28:23 am] * Not syncing Image between group(s)
[02-27-16 11:28:23 am] | Image Name: Fedora22LivingRoom
[02-27-16 11:28:23 am] | I am the only member
[02-27-16 11:28:23 am] * Not syncing Image between group(s)
[02-27-16 11:28:23 am] | Image Name: Fedora23LivingRoom
[02-27-16 11:28:23 am] | I am the only member
[02-27-16 11:28:23 am] * Not syncing Image between group(s)
[02-27-16 11:28:23 am] | Image Name: Win7
[02-27-16 11:28:23 am] | I am the only member
[02-27-16 11:28:23 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:28:23 am] | Image name: CentOS7Optiplex745UpdateBASE
[02-27-16 11:28:24 am] * Starting Sync Actions
[02-27-16 11:28:24 am] * Started sync for Image CentOS7Optiplex745UpdateBASE
[02-27-16 11:28:24 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:28:24 am] | Image name: Fedora22LivingRoom
[02-27-16 11:28:24 am] * Starting Sync Actions
[02-27-16 11:28:24 am] * Started sync for Image Fedora22LivingRoom
[02-27-16 11:28:24 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:28:24 am] | Image name: Fedora23LivingRoom
[02-27-16 11:28:24 am] * Starting Sync Actions
[02-27-16 11:28:24 am] * Started sync for Image Fedora23LivingRoom
[02-27-16 11:28:24 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:28:24 am] | Image name: Win7
[02-27-16 11:28:25 am] * Starting Sync Actions
[02-27-16 11:28:25 am] * Started sync for Image Win7
[02-27-16 11:38:25 am] * Starting Image Replication.
[02-27-16 11:38:25 am] * We are group ID: #1
[02-27-16 11:38:25 am] | We are group name: default
[02-27-16 11:38:25 am] * We have node ID: #1
[02-27-16 11:38:25 am] | We are node name: Master
[02-27-16 11:38:25 am] * Not syncing Image between group(s)
[02-27-16 11:38:25 am] | Image Name: CentOS7Optiplex745UpdateBASE
[02-27-16 11:38:25 am] | I am the only member
[02-27-16 11:38:25 am] * Not syncing Image between group(s)
[02-27-16 11:38:25 am] | Image Name: Fedora22LivingRoom
[02-27-16 11:38:25 am] | I am the only member
[02-27-16 11:38:25 am] * Not syncing Image between group(s)
[02-27-16 11:38:25 am] | Image Name: Fedora23LivingRoom
[02-27-16 11:38:25 am] | I am the only member
[02-27-16 11:38:25 am] * Not syncing Image between group(s)
[02-27-16 11:38:25 am] | Image Name: Win7
[02-27-16 11:38:25 am] | I am the only member
[02-27-16 11:38:25 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:38:25 am] | Image name: CentOS7Optiplex745UpdateBASE
[02-27-16 11:38:25 am] * Starting Sync Actions
[02-27-16 11:38:25 am] * Started sync for Image CentOS7Optiplex745UpdateBASE
[02-27-16 11:38:25 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:38:25 am] | Image name: Fedora22LivingRoom
[02-27-16 11:38:26 am] * Starting Sync Actions
[02-27-16 11:38:26 am] * Started sync for Image Fedora22LivingRoom
[02-27-16 11:38:26 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:38:26 am] | Image name: Fedora23LivingRoom
[02-27-16 11:38:26 am] * Starting Sync Actions
[02-27-16 11:38:26 am] * Started sync for Image Fedora23LivingRoom
[02-27-16 11:38:26 am] * Found Image to transfer to 2 node(s)
[02-27-16 11:38:26 am] | Image name: Win7
[02-27-16 11:38:26 am] * Starting Sync Actions
[02-27-16 11:38:26 am] * Started sync for Image Win7

I decided, ok, time to deploy and see how it goes. So I deployed the image. I noticed that the sending server was the slave node, according to the bandwidth charts.

And after the image was deployed, I booted it up and after a while, a notice at the top of my Fedora 23 desktop appeared, saying updates were ready to be installed!! How interesting!

So, I opened a terminal, switched to root, and ran dnf update. There were 600ish MB of updates available.

Replication did not happen properly.

Tom Elliott

I can’t see that the the replication did not happen properly. Can you get the actual transfer log for the node that the image deployed from?

Wayne Workman

@Tom-Elliott Sure can, that is in /opt/fog/log ?

Tom Elliott

@Wayne-Workman yes and it should be in the log viewer as well

Wayne Workman

Here is the unique SHA1 checksums of all files on both servers that have undergone change:

FIle list

Wayne Workman

Here is what’s from the first one (there are two called “Slave” in there):

Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.fixed_size_partitions' (2 B/s)
Transferring file `d1.has_grub'
Finished transfer `d1.has_grub'
Transferring file `d1.mbr'
Finished transfer `d1.mbr' (936.6 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.minimum.partitions' (184 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.original.fstypes' (33 B/s)
Transferring file `d1.original.swapuuids'
Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.original.swapuuids'
Transferring file `d1.partitions'
Finished transfer `d1.fixed_size_partitions' (2 B/s)
Transferring file `d1.has_grub'
Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.partitions' (192 B/s)
Transferring file `d1p1.img'
Finished transfer `d1.has_grub'
Transferring file `d1.mbr'
Finished transfer `d1.fixed_size_partitions' (4 B/s)
Transferring file `d1.has_grub'
Finished transfer `d1.has_grub'
Transferring file `d1.mbr'
Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.fixed_size_partitions' (2 B/s)
Transferring file `d1.mbr'
Finished transfer `d1.mbr' (905.0 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.mbr' (921.7 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.minimum.partitions' (184 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.minimum.partitions' (310 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.original.fstypes' (31 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.original.fstypes' (80 B/s)
Transferring file `d1.original.partitions'
Finished transfer `d1.original.swapuuids' (47 B/s)
Transferring file `d1.partitions'
Finished transfer `d1.partitions' (192 B/s)
Transferring file `d1p1.img'
Finished transfer `d1.original.partitions' (310 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.mbr' (542.1 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.original.swapuuids' (47 B/s)
Transferring file `d1p1.img'
Finished transfer `d1.minimum.partitions' (184 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.original.fstypes' (30 B/s)
Transferring file `d1.original.partitions'
Finished transfer `d1.original.partitions' (190 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.original.swapuuids'
Transferring file `d1p1.img'
Finished transfer `d1p1.img' (992.8 KiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p1.img' (570.3 KiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p1.img' (798.3 KiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p1.img' (827.2 KiB/s)
Finished transfer `d1p2.img' (930.2 KiB/s)
Finished transfer `d1p2.img' (3.27 MiB/s)
Transferring file `d1p4.img'
Finished transfer `d1p4.img' (127 B/s)
Transferring file `d1p5.img'
Finished transfer `d1p5.img' (2.43 MiB/s)
Finished transfer `d1p2.img' (10.14 MiB/s)
Removing old file `d1.original.partitions'
Removing old file `d1.fixed_size_partitions'
Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.fixed_size_partitions' (1 B/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.minimum.partitions' (6 B/s)
Transferring file `d1.partitions'
Finished transfer `d1.partitions' (190 B/s)
Transferring file `d1p1.img'
Finished transfer `d1p1.img' (3.0 KiB/s)
Removing old file `d1p2.img'
Transferring file `d1p2.img'
Finished transfer `d1p2.img' (12.91 MiB/s)
Transferring file `d1.fixed_size_partitions'
Transferring file `d1.fixed_size_partitions'
Transferring file `d1.fixed_size_partitions'
Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.fixed_size_partitions' (4 B/s)
Transferring file `d1.has_grub'
Finished transfer `d1.fixed_size_partitions' (2 B/s)
Transferring file `d1.has_grub'
Finished transfer `d1.fixed_size_partitions' (1 B/s)
Transferring file `d1.mbr'
Finished transfer `d1.fixed_size_partitions' (2 B/s)
Transferring file `d1.has_grub'
Finished transfer `d1.has_grub'
Finished transfer `d1.has_grub'
Transferring file `d1.mbr'
Transferring file `d1.mbr'
Finished transfer `d1.has_grub'
Transferring file `d1.mbr'
Finished transfer `d1.mbr' (873.0 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.minimum.partitions' (190 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.original.fstypes' (30 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.original.swapuuids'
Transferring file `d1.partitions'
Finished transfer `d1.partitions' (190 B/s)
Transferring file `d1p1.img'
Finished transfer `d1.mbr' (930.3 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.minimum.partitions' (310 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.mbr' (818.7 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.original.fstypes' (80 B/s)
Transferring file `d1.original.partitions'
Finished transfer `d1.original.partitions' (310 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.minimum.partitions' (184 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.mbr' (895.8 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.original.swapuuids' (47 B/s)
Transferring file `d1p1.img'
Finished transfer `d1.original.fstypes' (33 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.original.swapuuids'
Transferring file `d1.partitions'
Finished transfer `d1.minimum.partitions' (184 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.partitions' (192 B/s)
Transferring file `d1p1.img'
Finished transfer `d1.original.fstypes' (31 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.original.swapuuids' (47 B/s)
Transferring file `d1.partitions'
Finished transfer `d1.partitions' (192 B/s)
Transferring file `d1p1.img'
Finished transfer `d1p1.img' (2.53 MiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p1.img' (1.48 MiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p1.img' (770.6 KiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p2.img' (1.72 MiB/s)
Finished transfer `d1p1.img' (1.51 MiB/s)
Finished transfer `d1p2.img' (2.49 MiB/s)
Transferring file `d1p4.img'
Finished transfer `d1p4.img' (127 B/s)
Transferring file `d1p5.img'
Finished transfer `d1p5.img' (1.95 MiB/s)
Finished transfer `d1p2.img' (7.16 MiB/s)
Transferring file `d1.fixed_size_partitions'
Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.fixed_size_partitions' (2 B/s)
Transferring file `d1.has_grub'
Finished transfer `d1.fixed_size_partitions' (2 B/s)
Transferring file `d1.has_grub'
Finished transfer `d1.has_grub'
Transferring file `d1.mbr'
Finished transfer `d1.has_grub'
Transferring file `d1.mbr'
Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.mbr' (821.9 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.fixed_size_partitions' (4 B/s)
Transferring file `d1.has_grub'
Finished transfer `d1.has_grub'
Transferring file `d1.mbr'
Finished transfer `d1.minimum.partitions' (184 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.mbr' (892.0 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.original.fstypes' (31 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.minimum.partitions' (184 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.original.swapuuids' (47 B/s)
Transferring file `d1.partitions'
Finished transfer `d1.partitions' (192 B/s)
Transferring file `d1p1.img'
Finished transfer `d1.original.fstypes' (33 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.original.swapuuids'
Transferring file `d1.partitions'
Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.partitions' (192 B/s)
Transferring file `d1p1.img'
Finished transfer `d1.fixed_size_partitions' (1 B/s)
Transferring file `d1.mbr'
Finished transfer `d1.mbr' (919.7 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.minimum.partitions' (310 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.mbr' (816.3 KiB/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.minimum.partitions' (190 B/s)
Transferring file `d1.original.fstypes'
Finished transfer `d1.original.fstypes' (80 B/s)
Transferring file `d1.original.partitions'
Finished transfer `d1.original.fstypes' (30 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.original.swapuuids'
Transferring file `d1.partitions'
Finished transfer `d1.original.partitions' (310 B/s)
Transferring file `d1.original.swapuuids'
Finished transfer `d1.partitions' (190 B/s)
Transferring file `d1p1.img'
Finished transfer `d1.original.swapuuids' (47 B/s)
Transferring file `d1p1.img'
Finished transfer `d1p1.img' (1.60 MiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p1.img' (1.84 MiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p1.img' (1.26 MiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p1.img' (1.40 MiB/s)
Finished transfer `d1p2.img' (1.98 MiB/s)
Finished transfer `d1p2.img' (2.83 MiB/s)
Transferring file `d1p4.img'
Finished transfer `d1p4.img' (127 B/s)
Transferring file `d1p5.img'
Finished transfer `d1p5.img' (2.24 MiB/s)
Finished transfer `d1p2.img' (6.78 MiB/s)
Removing old file `d1.original.partitions'
Removing old file `d1p4.img'
Transferring file `d1.fixed_size_partitions'
Finished transfer `d1.fixed_size_partitions' (2 B/s)
Transferring file `d1.minimum.partitions'
Finished transfer `d1.minimum.partitions' (58 B/s)
Removing old file `d1.original.fstypes'
Transferring file `d1.original.fstypes'
Finished transfer `d1.original.fstypes' (48 B/s)
Transferring file `d1.partitions'
Finished transfer `d1.partitions' (368 B/s)
Transferring file `d1p1.img'
Finished transfer `d1p1.img' (9.12 MiB/s)
Transferring file `d1p2.img'
Finished transfer `d1p2.img' (6.27 MiB/s)
Transferring file `d1p4.ebr'
Finished transfer `d1p4.ebr' (512 B/s)
Transferring file `d1p5.ebr'
Finished transfer `d1p5.ebr' (512 B/s)
Transferring file `d1p5.img'
Finished transfer `d1p5.img' (48.5 KiB/s)

It’s tough to see what’s going on when there are no dates and no full file paths.

This is a bit concerning, also, note the scroll bar…

0_1456600624478_snapin replication.png

Wayne Workman

The checksums on the slave node are saying that partitions d1p1, d1p2, and d1p5 were not changed after replication completed.

Wayne Workman

MariaDB [fog]> select * from fileChecksums;
+-----------------+----------+------------+------------------------------------------+---------------------------------------------------------------+
| fileChecksumsID | fileHost | fileTime   | fileSum                                  | fileLocation                                                  |
+-----------------+----------+------------+------------------------------------------+---------------------------------------------------------------+
|               1 | 10.0.0.4 | 1456554030 | 7d06f3cc1fc461ef98864ed307742fd1ddf76be1 | /images/postdownloadscripts/fog.postdownload                  |
|               2 | 10.0.0.4 | 1456554030 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/dev/.mntcheck                                         |
|               3 | 10.0.0.4 | 1456554030 | e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e | /images/CentOS7Optiplex745UpdateBASE/d1.fixed_size_partitions |
|               4 | 10.0.0.4 | 1456554030 | a910a281b36b70a27be8b7752ffe52151bf8b013 | /images/CentOS7Optiplex745UpdateBASE/d1.partitions            |
|               5 | 10.0.0.4 | 1456554031 | a2d9ccb085562903966212cd08b5b21c63d8754a | /images/CentOS7Optiplex745UpdateBASE/d1.original.fstypes      |
|               6 | 10.0.0.4 | 1456554031 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.original.swapuuids    |
|               7 | 10.0.0.4 | 1456554031 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.has_grub              |
|               8 | 10.0.0.4 | 1456554031 | 4a0e35d60c41d45db95a5168516c9ebea638c41b | /images/CentOS7Optiplex745UpdateBASE/d1.mbr                   |
|               9 | 10.0.0.4 | 1456554031 | de5c8b5deb20cdf3ff4ae685993e6eaff40eb369 | /images/CentOS7Optiplex745UpdateBASE/d1.minimum.partitions    |
|              10 | 10.0.0.4 | 1456554076 | 21d368ce5f8398bdcccf37f9d9479b6c8d5893b8 | /images/CentOS7Optiplex745UpdateBASE/d1p1.img                 |
|              11 | 10.0.0.4 | 1456554124 | 838381c5d3c2071c55c6c3f57c701c62a02abf0d | /images/CentOS7Optiplex745UpdateBASE/d1p2.img                 |
|              12 | 10.0.0.4 | 1456554125 | 7448d8798a4380162d4b56f9b452e2f6f9e24e7a | /images/Fedora22LivingRoom/d1.fixed_size_partitions           |
|              13 | 10.0.0.4 | 1456554125 | e383ef7b4700bd13487ec8474389ec62dfb1964f | /images/Fedora22LivingRoom/d1.partitions                      |
|              14 | 10.0.0.4 | 1456554125 | 4fb695c407928af518e7f7820ca5066a5a386c42 | /images/Fedora22LivingRoom/d1.original.fstypes                |
|              15 | 10.0.0.4 | 1456554125 | 745c70e3928ffbf2a4ac0cd83a49ce641f76d7c5 | /images/Fedora22LivingRoom/d1.original.swapuuids              |
|              16 | 10.0.0.4 | 1456554125 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora22LivingRoom/d1.has_grub                        |
|              17 | 10.0.0.4 | 1456554125 | 23d9ad9b43a88f0f7159314c8b02ce0aa7db402f | /images/Fedora22LivingRoom/d1.mbr                             |
|              18 | 10.0.0.4 | 1456554125 | 034ab561181f7137919db2ce24c49a3ee4fba7da | /images/Fedora22LivingRoom/d1.minimum.partitions              |
|              19 | 10.0.0.4 | 1456554230 | 3e89bd52a412e796bb1d47f35ed5efe71d234cb2 | /images/Fedora22LivingRoom/d1p1.img                           |
|              20 | 10.0.0.4 | 1456554231 | ccd1afaabd987136c3be65bd604b6b230c14c92e | /images/Fedora23LivingRoom/d1.fixed_size_partitions           |
|              21 | 10.0.0.4 | 1456554231 | 861802c73c9400c5de4c81b71fbbcdfb3b131201 | /images/Fedora23LivingRoom/d1.original.partitions             |
|              22 | 10.0.0.4 | 1456554231 | 5cdab89de5dd88254a298d2f4badd9e4ebf0ee50 | /images/Fedora23LivingRoom/d1.original.fstypes                |
|              23 | 10.0.0.4 | 1456554231 | a3929b85c1cce6b9934db527550068ccef11f8fd | /images/Fedora23LivingRoom/d1.original.swapuuids              |
|              24 | 10.0.0.4 | 1456554232 | 7eaa3c92068d8e308ee47728c72adcc35e774a8c | /images/Fedora23LivingRoom/d1.mbr                             |
|              25 | 10.0.0.4 | 1456554232 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora23LivingRoom/d1.has_grub                        |
|              26 | 10.0.0.4 | 1456554232 | 04561e6fb817deacbcbf9bd7cb54d29a1d0cccf0 | /images/Fedora23LivingRoom/d1.minimum.partitions              |
|              27 | 10.0.0.4 | 1456554236 | ffccd044af5c0295c8fce72c9db39a64b9b96d7a | /images/Fedora23LivingRoom/d1p1.img                           |
|              28 | 10.0.0.4 | 1456554321 | 2496d77b7dc4aa68e5d04b72e5f656461febcc69 | /images/Fedora23LivingRoom/d1p2.img                           |
|              29 | 10.0.0.4 | 1456554322 | 22015e84b062ed609e0f9e84f93d240fa20ff477 | /images/Fedora23LivingRoom/d1p4.img                           |
|              30 | 10.0.0.4 | 1456554323 | 98bfd22dbf59c8ba3398a2d7c542f265768e9675 | /images/Fedora23LivingRoom/d1p5.img                           |
|              31 | 10.0.0.4 | 1456554323 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/.mntcheck                                             |
|              32 | 10.0.0.4 | 1456554323 | adc83b19e793491b1c6ea0fd8b46cd9f32e592fc | /images/Win7/d1.fixed_size_partitions                         |
|              33 | 10.0.0.4 | 1456554324 | 3ed69cf742c34690cc061eb44d34482e87b90776 | /images/Win7/d1.partitions                                    |
|              34 | 10.0.0.4 | 1456554324 | d9bdc2da3027044ae124831e8cca1b2f534d7163 | /images/Win7/d1.original.fstypes                              |
|              35 | 10.0.0.4 | 1456554324 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Win7/d1.original.swapuuids                            |
|              36 | 10.0.0.4 | 1456554324 | 4a9712192b5b0c272cf0797fe033ab2f7956d79d | /images/Win7/d1.mbr                                           |
|              37 | 10.0.0.4 | 1456554324 | 139c5dbc9b92a04f0966c90d4fd5c1ff8a4a9eb1 | /images/Win7/d1.minimum.partitions                            |
|              38 | 10.0.0.4 | 1456554324 | d3e3953115255bdd8cedb03a6ada3a5a69f9c3b3 | /images/Win7/d1p1.img                                         |
|              39 | 10.0.0.4 | 1456554652 | a7f07cef8abbdaf0137cb8e1191e22e33f75d863 | /images/Win7/d1p2.img                                         |
|              40 | 10.0.0.4 | 1456554652 | f09f7644b27f9b6964c2f35028dfc406bdad22b6 | /opt/fog/snapins/ssl/.srvprivate.key                          |
|              41 | 10.0.0.4 | 1456554652 | 606a535989dfa30f585b0b2b96ff3e47d9df4dc7 | /opt/fog/snapins/ssl/fog.csr                                  |
|              42 | 10.0.0.4 | 1456554653 | d8dcc09a122213729f2de9937ea77d984ea01323 | /opt/fog/snapins/ssl/CA/.fogCA.key                            |
|              43 | 10.0.0.4 | 1456554653 | b0564bc10ecbedc66e61766c27bfdfacf7c508fa | /opt/fog/snapins/ssl/CA/.fogCA.pem                            |
|              44 | 10.0.0.4 | 1456554653 | 0a4b6296e1f413526bca330f50ed4661a2b5d381 | /opt/fog/snapins/ssl/CA/.srl                                  |
|              45 | 10.0.0.4 | 1456554654 | d92e5813b41b1e24c81dcc8ac9efa66f27472c70 | /opt/fog/snapins/googlechromestandaloneenterprise.msi         |
|              46 | 10.0.0.5 | 1456554727 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/.mntcheck                                             |
|              47 | 10.0.0.5 | 1456554727 | 7d06f3cc1fc461ef98864ed307742fd1ddf76be1 | /images/postdownloadscripts/fog.postdownload                  |
|              48 | 10.0.0.5 | 1456554727 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/dev/.mntcheck                                         |
|              49 | 10.0.0.5 | 1456554727 | e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e | /images/CentOS7Optiplex745UpdateBASE/d1.fixed_size_partitions |
|              50 | 10.0.0.5 | 1456554728 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.has_grub              |
|              51 | 10.0.0.5 | 1456554728 | 4a0e35d60c41d45db95a5168516c9ebea638c41b | /images/CentOS7Optiplex745UpdateBASE/d1.mbr                   |
|              52 | 10.0.0.5 | 1456554728 | de5c8b5deb20cdf3ff4ae685993e6eaff40eb369 | /images/CentOS7Optiplex745UpdateBASE/d1.minimum.partitions    |
|              53 | 10.0.0.5 | 1456554728 | a2d9ccb085562903966212cd08b5b21c63d8754a | /images/CentOS7Optiplex745UpdateBASE/d1.original.fstypes      |
|              54 | 10.0.0.5 | 1456554728 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.original.swapuuids    |
|              55 | 10.0.0.5 | 1456554728 | a910a281b36b70a27be8b7752ffe52151bf8b013 | /images/CentOS7Optiplex745UpdateBASE/d1.partitions            |
|              56 | 10.0.0.5 | 1456554763 | 21d368ce5f8398bdcccf37f9d9479b6c8d5893b8 | /images/CentOS7Optiplex745UpdateBASE/d1p1.img                 |
|              57 | 10.0.0.5 | 1456554831 | 838381c5d3c2071c55c6c3f57c701c62a02abf0d | /images/CentOS7Optiplex745UpdateBASE/d1p2.img                 |
|              58 | 10.0.0.5 | 1456554833 | 7448d8798a4380162d4b56f9b452e2f6f9e24e7a | /images/Fedora22LivingRoom/d1.fixed_size_partitions           |
|              59 | 10.0.0.5 | 1456554833 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora22LivingRoom/d1.has_grub                        |
|              60 | 10.0.0.5 | 1456554833 | 23d9ad9b43a88f0f7159314c8b02ce0aa7db402f | /images/Fedora22LivingRoom/d1.mbr                             |
|              61 | 10.0.0.5 | 1456554833 | 034ab561181f7137919db2ce24c49a3ee4fba7da | /images/Fedora22LivingRoom/d1.minimum.partitions              |
|              62 | 10.0.0.5 | 1456554833 | 4fb695c407928af518e7f7820ca5066a5a386c42 | /images/Fedora22LivingRoom/d1.original.fstypes                |
|              63 | 10.0.0.5 | 1456554833 | 745c70e3928ffbf2a4ac0cd83a49ce641f76d7c5 | /images/Fedora22LivingRoom/d1.original.swapuuids              |
|              64 | 10.0.0.5 | 1456554833 | e383ef7b4700bd13487ec8474389ec62dfb1964f | /images/Fedora22LivingRoom/d1.partitions                      |
|              65 | 10.0.0.5 | 1456554939 | 3e89bd52a412e796bb1d47f35ed5efe71d234cb2 | /images/Fedora22LivingRoom/d1p1.img                           |
|              66 | 10.0.0.5 | 1456554940 | ccd1afaabd987136c3be65bd604b6b230c14c92e | /images/Fedora23LivingRoom/d1.fixed_size_partitions           |
|              67 | 10.0.0.5 | 1456554940 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora23LivingRoom/d1.has_grub                        |
|              68 | 10.0.0.5 | 1456554940 | 7eaa3c92068d8e308ee47728c72adcc35e774a8c | /images/Fedora23LivingRoom/d1.mbr                             |
|              69 | 10.0.0.5 | 1456554940 | 04561e6fb817deacbcbf9bd7cb54d29a1d0cccf0 | /images/Fedora23LivingRoom/d1.minimum.partitions              |
|              70 | 10.0.0.5 | 1456554940 | 5cdab89de5dd88254a298d2f4badd9e4ebf0ee50 | /images/Fedora23LivingRoom/d1.original.fstypes                |
|              71 | 10.0.0.5 | 1456554940 | 861802c73c9400c5de4c81b71fbbcdfb3b131201 | /images/Fedora23LivingRoom/d1.original.partitions             |
|              72 | 10.0.0.5 | 1456554940 | a3929b85c1cce6b9934db527550068ccef11f8fd | /images/Fedora23LivingRoom/d1.original.swapuuids              |
|              73 | 10.0.0.5 | 1456554945 | ffccd044af5c0295c8fce72c9db39a64b9b96d7a | /images/Fedora23LivingRoom/d1p1.img                           |
|              74 | 10.0.0.5 | 1456555023 | 2496d77b7dc4aa68e5d04b72e5f656461febcc69 | /images/Fedora23LivingRoom/d1p2.img                           |
|              75 | 10.0.0.5 | 1456555024 | 22015e84b062ed609e0f9e84f93d240fa20ff477 | /images/Fedora23LivingRoom/d1p4.img                           |
|              76 | 10.0.0.5 | 1456555025 | 98bfd22dbf59c8ba3398a2d7c542f265768e9675 | /images/Fedora23LivingRoom/d1p5.img                           |
|              77 | 10.0.0.5 | 1456555025 | adc83b19e793491b1c6ea0fd8b46cd9f32e592fc | /images/Win7/d1.fixed_size_partitions                         |
|              78 | 10.0.0.5 | 1456555025 | 4a9712192b5b0c272cf0797fe033ab2f7956d79d | /images/Win7/d1.mbr                                           |
|              79 | 10.0.0.5 | 1456555026 | 139c5dbc9b92a04f0966c90d4fd5c1ff8a4a9eb1 | /images/Win7/d1.minimum.partitions                            |
|              80 | 10.0.0.5 | 1456555026 | d9bdc2da3027044ae124831e8cca1b2f534d7163 | /images/Win7/d1.original.fstypes                              |
|              81 | 10.0.0.5 | 1456555026 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Win7/d1.original.swapuuids                            |
|              82 | 10.0.0.5 | 1456555026 | 3ed69cf742c34690cc061eb44d34482e87b90776 | /images/Win7/d1.partitions                                    |
|              83 | 10.0.0.5 | 1456555026 | d3e3953115255bdd8cedb03a6ada3a5a69f9c3b3 | /images/Win7/d1p1.img                                         |
|              84 | 10.0.0.5 | 1456555263 | a7f07cef8abbdaf0137cb8e1191e22e33f75d863 | /images/Win7/d1p2.img                                         |
|              85 | 10.0.0.5 | 1456555264 | f09f7644b27f9b6964c2f35028dfc406bdad22b6 | /opt/fog/snapins/ssl/.srvprivate.key                          |
|              86 | 10.0.0.5 | 1456555264 | 606a535989dfa30f585b0b2b96ff3e47d9df4dc7 | /opt/fog/snapins/ssl/fog.csr                                  |
|              87 | 10.0.0.5 | 1456555264 | d8dcc09a122213729f2de9937ea77d984ea01323 | /opt/fog/snapins/ssl/CA/.fogCA.key                            |
|              88 | 10.0.0.5 | 1456555264 | b0564bc10ecbedc66e61766c27bfdfacf7c508fa | /opt/fog/snapins/ssl/CA/.fogCA.pem                            |
|              89 | 10.0.0.5 | 1456555264 | 15c32736d8e14df26be33c8d12e0dc0539d43822 | /opt/fog/snapins/ssl/CA/.srl                                  |
|              90 | 10.0.0.4 | 1456596538 | 7d06f3cc1fc461ef98864ed307742fd1ddf76be1 | /images/postdownloadscripts/fog.postdownload                  |
|              91 | 10.0.0.4 | 1456596538 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/dev/.mntcheck                                         |
|              92 | 10.0.0.4 | 1456596538 | e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e | /images/CentOS7Optiplex745UpdateBASE/d1.fixed_size_partitions |
|              93 | 10.0.0.4 | 1456596538 | a910a281b36b70a27be8b7752ffe52151bf8b013 | /images/CentOS7Optiplex745UpdateBASE/d1.partitions            |
|              94 | 10.0.0.4 | 1456596538 | a2d9ccb085562903966212cd08b5b21c63d8754a | /images/CentOS7Optiplex745UpdateBASE/d1.original.fstypes      |
|              95 | 10.0.0.4 | 1456596538 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.original.swapuuids    |
|              96 | 10.0.0.4 | 1456596538 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.has_grub              |
|              97 | 10.0.0.4 | 1456596538 | 4a0e35d60c41d45db95a5168516c9ebea638c41b | /images/CentOS7Optiplex745UpdateBASE/d1.mbr                   |
|              98 | 10.0.0.4 | 1456596538 | de5c8b5deb20cdf3ff4ae685993e6eaff40eb369 | /images/CentOS7Optiplex745UpdateBASE/d1.minimum.partitions    |
|              99 | 10.0.0.4 | 1456596568 | 21d368ce5f8398bdcccf37f9d9479b6c8d5893b8 | /images/CentOS7Optiplex745UpdateBASE/d1p1.img                 |
|             100 | 10.0.0.4 | 1456596612 | 838381c5d3c2071c55c6c3f57c701c62a02abf0d | /images/CentOS7Optiplex745UpdateBASE/d1p2.img                 |
|             101 | 10.0.0.4 | 1456596613 | 7448d8798a4380162d4b56f9b452e2f6f9e24e7a | /images/Fedora22LivingRoom/d1.fixed_size_partitions           |
|             102 | 10.0.0.4 | 1456596613 | e383ef7b4700bd13487ec8474389ec62dfb1964f | /images/Fedora22LivingRoom/d1.partitions                      |
|             103 | 10.0.0.4 | 1456596613 | 4fb695c407928af518e7f7820ca5066a5a386c42 | /images/Fedora22LivingRoom/d1.original.fstypes                |
|             104 | 10.0.0.4 | 1456596613 | 745c70e3928ffbf2a4ac0cd83a49ce641f76d7c5 | /images/Fedora22LivingRoom/d1.original.swapuuids              |
|             105 | 10.0.0.4 | 1456596613 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora22LivingRoom/d1.has_grub                        |
|             106 | 10.0.0.4 | 1456596614 | 23d9ad9b43a88f0f7159314c8b02ce0aa7db402f | /images/Fedora22LivingRoom/d1.mbr                             |
|             107 | 10.0.0.4 | 1456596614 | 034ab561181f7137919db2ce24c49a3ee4fba7da | /images/Fedora22LivingRoom/d1.minimum.partitions              |
|             108 | 10.0.0.4 | 1456596681 | 3e89bd52a412e796bb1d47f35ed5efe71d234cb2 | /images/Fedora22LivingRoom/d1p1.img                           |
|             109 | 10.0.0.4 | 1456596682 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/.mntcheck                                             |
|             110 | 10.0.0.4 | 1456596682 | adc83b19e793491b1c6ea0fd8b46cd9f32e592fc | /images/Win7/d1.fixed_size_partitions                         |
|             111 | 10.0.0.4 | 1456596682 | 3ed69cf742c34690cc061eb44d34482e87b90776 | /images/Win7/d1.partitions                                    |
|             112 | 10.0.0.4 | 1456596682 | d9bdc2da3027044ae124831e8cca1b2f534d7163 | /images/Win7/d1.original.fstypes                              |
|             113 | 10.0.0.4 | 1456596682 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Win7/d1.original.swapuuids                            |
|             114 | 10.0.0.4 | 1456596682 | 4a9712192b5b0c272cf0797fe033ab2f7956d79d | /images/Win7/d1.mbr                                           |
|             115 | 10.0.0.4 | 1456596682 | 139c5dbc9b92a04f0966c90d4fd5c1ff8a4a9eb1 | /images/Win7/d1.minimum.partitions                            |
|             116 | 10.0.0.4 | 1456596683 | d3e3953115255bdd8cedb03a6ada3a5a69f9c3b3 | /images/Win7/d1p1.img                                         |
|             117 | 10.0.0.4 | 1456597082 | a7f07cef8abbdaf0137cb8e1191e22e33f75d863 | /images/Win7/d1p2.img                                         |
|             118 | 10.0.0.4 | 1456597086 | b40761a713f76d25e4d002eaf84ef2c5d1a79751 | /images/Fedora23LivingRoom/d1.fixed_size_partitions           |
|             119 | 10.0.0.4 | 1456597087 | 950c35c0634b8c6eef5c8ffac48112b5557a0ef6 | /images/Fedora23LivingRoom/d1.partitions                      |
|             120 | 10.0.0.4 | 1456597087 | 74453471faea8f6ba64880b066e79cc5ad175693 | /images/Fedora23LivingRoom/d1.original.fstypes                |
|             121 | 10.0.0.4 | 1456597087 | 337d3942eb95bf3acbbb8f445752183f3f484e0e | /images/Fedora23LivingRoom/d1.original.swapuuids              |
|             122 | 10.0.0.4 | 1456597087 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora23LivingRoom/d1.has_grub                        |
|             123 | 10.0.0.4 | 1456597087 | 68e325611ffaf97a8256c00158ed8c5ebb58f50d | /images/Fedora23LivingRoom/d1.mbr                             |
|             124 | 10.0.0.4 | 1456597087 | 151ddf4da950213ec9ba8cf22c0bad21e11e83d4 | /images/Fedora23LivingRoom/d1.minimum.partitions              |
|             125 | 10.0.0.4 | 1456597087 | 5dc2ff987315fb208d54ac2ce473af7f46e77533 | /images/Fedora23LivingRoom/d1p4.ebr                           |
|             126 | 10.0.0.4 | 1456597087 | 5c3eb80066420002bc3dcc7ca4ab6efad7ed4ae5 | /images/Fedora23LivingRoom/d1p5.ebr                           |
|             127 | 10.0.0.4 | 1456597092 | 22f68eb0f40c50279048121f2e521099152f0d29 | /images/Fedora23LivingRoom/d1p1.img                           |
|             128 | 10.0.0.4 | 1456597232 | 6b671edc75587fa51750c5f1d8c71e7383df0c87 | /images/Fedora23LivingRoom/d1p2.img                           |
|             129 | 10.0.0.4 | 1456597234 | 06672a79cfdf1b66552f33d306f21cd286a45567 | /images/Fedora23LivingRoom/d1p5.img                           |
|             130 | 10.0.0.4 | 1456597235 | f09f7644b27f9b6964c2f35028dfc406bdad22b6 | /opt/fog/snapins/ssl/.srvprivate.key                          |
|             131 | 10.0.0.4 | 1456597235 | 606a535989dfa30f585b0b2b96ff3e47d9df4dc7 | /opt/fog/snapins/ssl/fog.csr                                  |
|             132 | 10.0.0.4 | 1456597235 | d8dcc09a122213729f2de9937ea77d984ea01323 | /opt/fog/snapins/ssl/CA/.fogCA.key                            |
|             133 | 10.0.0.4 | 1456597235 | b0564bc10ecbedc66e61766c27bfdfacf7c508fa | /opt/fog/snapins/ssl/CA/.fogCA.pem                            |
|             134 | 10.0.0.4 | 1456597235 | c6d47b64b1a6ff598a84ca596a894375956d4371 | /opt/fog/snapins/ssl/CA/.srl                                  |
|             135 | 10.0.0.4 | 1456597237 | d92e5813b41b1e24c81dcc8ac9efa66f27472c70 | /opt/fog/snapins/googlechromestandaloneenterprise.msi         |
|             136 | 10.0.0.5 | 1456597523 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/.mntcheck                                             |
|             137 | 10.0.0.5 | 1456597523 | 7d06f3cc1fc461ef98864ed307742fd1ddf76be1 | /images/postdownloadscripts/fog.postdownload                  |
|             138 | 10.0.0.5 | 1456597523 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/dev/.mntcheck                                         |
|             139 | 10.0.0.5 | 1456597523 | e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e | /images/CentOS7Optiplex745UpdateBASE/d1.fixed_size_partitions |
|             140 | 10.0.0.5 | 1456597523 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.has_grub              |
|             141 | 10.0.0.5 | 1456597523 | 4a0e35d60c41d45db95a5168516c9ebea638c41b | /images/CentOS7Optiplex745UpdateBASE/d1.mbr                   |
|             142 | 10.0.0.5 | 1456597523 | de5c8b5deb20cdf3ff4ae685993e6eaff40eb369 | /images/CentOS7Optiplex745UpdateBASE/d1.minimum.partitions    |
|             143 | 10.0.0.5 | 1456597523 | a2d9ccb085562903966212cd08b5b21c63d8754a | /images/CentOS7Optiplex745UpdateBASE/d1.original.fstypes      |
|             144 | 10.0.0.5 | 1456597523 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.original.swapuuids    |
|             145 | 10.0.0.5 | 1456597523 | a910a281b36b70a27be8b7752ffe52151bf8b013 | /images/CentOS7Optiplex745UpdateBASE/d1.partitions            |
|             146 | 10.0.0.5 | 1456597556 | 21d368ce5f8398bdcccf37f9d9479b6c8d5893b8 | /images/CentOS7Optiplex745UpdateBASE/d1p1.img                 |
|             147 | 10.0.0.5 | 1456597605 | 838381c5d3c2071c55c6c3f57c701c62a02abf0d | /images/CentOS7Optiplex745UpdateBASE/d1p2.img                 |
|             148 | 10.0.0.5 | 1456597606 | 7448d8798a4380162d4b56f9b452e2f6f9e24e7a | /images/Fedora22LivingRoom/d1.fixed_size_partitions           |
|             149 | 10.0.0.5 | 1456597606 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora22LivingRoom/d1.has_grub                        |
|             150 | 10.0.0.5 | 1456597606 | 23d9ad9b43a88f0f7159314c8b02ce0aa7db402f | /images/Fedora22LivingRoom/d1.mbr                             |
|             151 | 10.0.0.5 | 1456597606 | 034ab561181f7137919db2ce24c49a3ee4fba7da | /images/Fedora22LivingRoom/d1.minimum.partitions              |
|             152 | 10.0.0.5 | 1456597606 | 4fb695c407928af518e7f7820ca5066a5a386c42 | /images/Fedora22LivingRoom/d1.original.fstypes                |
|             153 | 10.0.0.5 | 1456597606 | 745c70e3928ffbf2a4ac0cd83a49ce641f76d7c5 | /images/Fedora22LivingRoom/d1.original.swapuuids              |
|             154 | 10.0.0.5 | 1456597606 | e383ef7b4700bd13487ec8474389ec62dfb1964f | /images/Fedora22LivingRoom/d1.partitions                      |
|             155 | 10.0.0.5 | 1456597685 | 3e89bd52a412e796bb1d47f35ed5efe71d234cb2 | /images/Fedora22LivingRoom/d1p1.img                           |
|             156 | 10.0.0.5 | 1456597686 | bd0125f265897b42f3a9c1cd065c26ac4d486443 | /images/Fedora23LivingRoom/d1.fixed_size_partitions           |
|             157 | 10.0.0.5 | 1456597686 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora23LivingRoom/d1.has_grub                        |
|             158 | 10.0.0.5 | 1456597686 | 7eaa3c92068d8e308ee47728c72adcc35e774a8c | /images/Fedora23LivingRoom/d1.mbr                             |
|             159 | 10.0.0.5 | 1456597686 | 0af0c2e070c0a318a5167484cb18f55703e49528 | /images/Fedora23LivingRoom/d1.minimum.partitions              |
|             160 | 10.0.0.5 | 1456597686 | a3929b85c1cce6b9934db527550068ccef11f8fd | /images/Fedora23LivingRoom/d1.original.swapuuids              |
|             161 | 10.0.0.5 | 1456597690 | 5cfd63fe1107d898c6588fff459e6dba0ecb6cce | /images/Fedora23LivingRoom/d1p1.img                           |
|             162 | 10.0.0.5 | 1456597797 | a4f5da5548d94ef13c260ce7069b1ce24ec43fed | /images/Fedora23LivingRoom/d1p2.img                           |
|             163 | 10.0.0.5 | 1456597798 | 62449efcb9b1142a77097f652b2f6148ecb9e70f | /images/Fedora23LivingRoom/d1p5.img                           |
|             164 | 10.0.0.5 | 1456597798 | 74453471faea8f6ba64880b066e79cc5ad175693 | /images/Fedora23LivingRoom/d1.original.fstypes                |
|             165 | 10.0.0.5 | 1456597799 | 950c35c0634b8c6eef5c8ffac48112b5557a0ef6 | /images/Fedora23LivingRoom/d1.partitions                      |
|             166 | 10.0.0.5 | 1456597799 | 5dc2ff987315fb208d54ac2ce473af7f46e77533 | /images/Fedora23LivingRoom/d1p4.ebr                           |
|             167 | 10.0.0.5 | 1456597799 | 5c3eb80066420002bc3dcc7ca4ab6efad7ed4ae5 | /images/Fedora23LivingRoom/d1p5.ebr                           |
|             168 | 10.0.0.5 | 1456597799 | adc83b19e793491b1c6ea0fd8b46cd9f32e592fc | /images/Win7/d1.fixed_size_partitions                         |
|             169 | 10.0.0.5 | 1456597799 | 4a9712192b5b0c272cf0797fe033ab2f7956d79d | /images/Win7/d1.mbr                                           |
|             170 | 10.0.0.5 | 1456597799 | 139c5dbc9b92a04f0966c90d4fd5c1ff8a4a9eb1 | /images/Win7/d1.minimum.partitions                            |
|             171 | 10.0.0.5 | 1456597799 | d9bdc2da3027044ae124831e8cca1b2f534d7163 | /images/Win7/d1.original.fstypes                              |
|             172 | 10.0.0.5 | 1456597799 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Win7/d1.original.swapuuids                            |
|             173 | 10.0.0.5 | 1456597799 | 3ed69cf742c34690cc061eb44d34482e87b90776 | /images/Win7/d1.partitions                                    |
|             174 | 10.0.0.5 | 1456597800 | d3e3953115255bdd8cedb03a6ada3a5a69f9c3b3 | /images/Win7/d1p1.img                                         |
|             175 | 10.0.0.5 | 1456598047 | a7f07cef8abbdaf0137cb8e1191e22e33f75d863 | /images/Win7/d1p2.img                                         |
|             176 | 10.0.0.5 | 1456598048 | f09f7644b27f9b6964c2f35028dfc406bdad22b6 | /opt/fog/snapins/ssl/.srvprivate.key                          |
|             177 | 10.0.0.5 | 1456598048 | 606a535989dfa30f585b0b2b96ff3e47d9df4dc7 | /opt/fog/snapins/ssl/fog.csr                                  |
|             178 | 10.0.0.5 | 1456598048 | d8dcc09a122213729f2de9937ea77d984ea01323 | /opt/fog/snapins/ssl/CA/.fogCA.key                            |
|             179 | 10.0.0.5 | 1456598048 | b0564bc10ecbedc66e61766c27bfdfacf7c508fa | /opt/fog/snapins/ssl/CA/.fogCA.pem                            |
|             180 | 10.0.0.5 | 1456598048 | 0a4b6296e1f413526bca330f50ed4661a2b5d381 | /opt/fog/snapins/ssl/CA/.srl                                  |
+-----------------+----------+------------+------------------------------------------+---------------------------------------------------------------+
180 rows in set (0.01 sec)

Wayne Workman

Look at these particular checksum records from the master and slave, taken before the image was uploaded, and after upload and after replication:

27 | 10.0.0.4 | 1456554236 | ffccd044af5c0295c8fce72c9db39a64b9b96d7a | /images/Fedora23LivingRoom/d1p1.img
73 | 10.0.0.5 | 1456554945 | ffccd044af5c0295c8fce72c9db39a64b9b96d7a | /images/Fedora23LivingRoom/d1p1.img

then capture a new image, wait for replication.... New checksums:

127 | 10.0.0.4 | 1456597092 | 22f68eb0f40c50279048121f2e521099152f0d29 | /images/Fedora23LivingRoom/d1p1.img
161 | 10.0.0.5 | 1456597690 | 5cfd63fe1107d898c6588fff459e6dba0ecb6cce | /images/Fedora23LivingRoom/d1p1.img

We can see that yes, these files were absolutely identical prior to the image being updated, on both nodes in the group.
After the image was updated, and after replication happened, we see that the files were changed, but are not identical.

Wayne Workman

If you look at my checksum records for my Win7 image from this thread, you’ll see that the same problem existed then too:
https://forums.fogproject.org/topic/6318/image-file-integrity/12

Only then, I didn’t “try out” my image so I was unaware of an issue and thought something was wrong with the logic I wrote, but I don’t think anything is wrong with it now. Looking at the actual data from both those tests and now, It’s clear there is a replication issue.

Wayne Workman

Yesterday evening, I was able to work with Tom in his test-bed to replicate the problem. Turns out, it’s a bug in lftp itself. Here’s the bug report that Tom found:
https://github.com/lavv17/lftp/issues/148

Fedora 23 ships with lftp version 4.6.1 so it has the bug. If the file being evaluated is smaller, it gets replaced correctly. If the file being evaluated is larger, lftp only adds the changed bits - resulting in a mostly old image that is highly corrupted.

Tom said he would work on using something else for replication in the mean time.

Tom Elliott

I added some preliminary checking to try to figure out if a file is the same. If it isn’t, it deletes it on the remote side, and then continues checking the other files. This means lftp should always start a fresh file.

Replication Failure

186

11.7k

17.1k

154.7k