Solved Replication Failure
-
@Tom-Elliott Sure can, that is in /opt/fog/log ?
-
@Wayne-Workman yes and it should be in the log viewer as well
-
Here is the unique SHA1 checksums of all files on both servers that have undergone change:
-
Here is what’s from the first one (there are two called “Slave” in there):
Transferring file `d1.fixed_size_partitions' Finished transfer `d1.fixed_size_partitions' (2 B/s) Transferring file `d1.has_grub' Finished transfer `d1.has_grub' Transferring file `d1.mbr' Finished transfer `d1.mbr' (936.6 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.minimum.partitions' (184 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.original.fstypes' (33 B/s) Transferring file `d1.original.swapuuids' Transferring file `d1.fixed_size_partitions' Finished transfer `d1.original.swapuuids' Transferring file `d1.partitions' Finished transfer `d1.fixed_size_partitions' (2 B/s) Transferring file `d1.has_grub' Transferring file `d1.fixed_size_partitions' Finished transfer `d1.partitions' (192 B/s) Transferring file `d1p1.img' Finished transfer `d1.has_grub' Transferring file `d1.mbr' Finished transfer `d1.fixed_size_partitions' (4 B/s) Transferring file `d1.has_grub' Finished transfer `d1.has_grub' Transferring file `d1.mbr' Transferring file `d1.fixed_size_partitions' Finished transfer `d1.fixed_size_partitions' (2 B/s) Transferring file `d1.mbr' Finished transfer `d1.mbr' (905.0 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.mbr' (921.7 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.minimum.partitions' (184 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.minimum.partitions' (310 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.original.fstypes' (31 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.original.fstypes' (80 B/s) Transferring file `d1.original.partitions' Finished transfer `d1.original.swapuuids' (47 B/s) Transferring file `d1.partitions' Finished transfer `d1.partitions' (192 B/s) Transferring file `d1p1.img' Finished transfer `d1.original.partitions' (310 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.mbr' (542.1 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.original.swapuuids' (47 B/s) Transferring file `d1p1.img' Finished transfer `d1.minimum.partitions' (184 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.original.fstypes' (30 B/s) Transferring file `d1.original.partitions' Finished transfer `d1.original.partitions' (190 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.original.swapuuids' Transferring file `d1p1.img' Finished transfer `d1p1.img' (992.8 KiB/s) Transferring file `d1p2.img' Finished transfer `d1p1.img' (570.3 KiB/s) Transferring file `d1p2.img' Finished transfer `d1p1.img' (798.3 KiB/s) Transferring file `d1p2.img' Finished transfer `d1p1.img' (827.2 KiB/s) Finished transfer `d1p2.img' (930.2 KiB/s) Finished transfer `d1p2.img' (3.27 MiB/s) Transferring file `d1p4.img' Finished transfer `d1p4.img' (127 B/s) Transferring file `d1p5.img' Finished transfer `d1p5.img' (2.43 MiB/s) Finished transfer `d1p2.img' (10.14 MiB/s) Removing old file `d1.original.partitions' Removing old file `d1.fixed_size_partitions' Transferring file `d1.fixed_size_partitions' Finished transfer `d1.fixed_size_partitions' (1 B/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.minimum.partitions' (6 B/s) Transferring file `d1.partitions' Finished transfer `d1.partitions' (190 B/s) Transferring file `d1p1.img' Finished transfer `d1p1.img' (3.0 KiB/s) Removing old file `d1p2.img' Transferring file `d1p2.img' Finished transfer `d1p2.img' (12.91 MiB/s) Transferring file `d1.fixed_size_partitions' Transferring file `d1.fixed_size_partitions' Transferring file `d1.fixed_size_partitions' Transferring file `d1.fixed_size_partitions' Finished transfer `d1.fixed_size_partitions' (4 B/s) Transferring file `d1.has_grub' Finished transfer `d1.fixed_size_partitions' (2 B/s) Transferring file `d1.has_grub' Finished transfer `d1.fixed_size_partitions' (1 B/s) Transferring file `d1.mbr' Finished transfer `d1.fixed_size_partitions' (2 B/s) Transferring file `d1.has_grub' Finished transfer `d1.has_grub' Finished transfer `d1.has_grub' Transferring file `d1.mbr' Transferring file `d1.mbr' Finished transfer `d1.has_grub' Transferring file `d1.mbr' Finished transfer `d1.mbr' (873.0 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.minimum.partitions' (190 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.original.fstypes' (30 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.original.swapuuids' Transferring file `d1.partitions' Finished transfer `d1.partitions' (190 B/s) Transferring file `d1p1.img' Finished transfer `d1.mbr' (930.3 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.minimum.partitions' (310 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.mbr' (818.7 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.original.fstypes' (80 B/s) Transferring file `d1.original.partitions' Finished transfer `d1.original.partitions' (310 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.minimum.partitions' (184 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.mbr' (895.8 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.original.swapuuids' (47 B/s) Transferring file `d1p1.img' Finished transfer `d1.original.fstypes' (33 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.original.swapuuids' Transferring file `d1.partitions' Finished transfer `d1.minimum.partitions' (184 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.partitions' (192 B/s) Transferring file `d1p1.img' Finished transfer `d1.original.fstypes' (31 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.original.swapuuids' (47 B/s) Transferring file `d1.partitions' Finished transfer `d1.partitions' (192 B/s) Transferring file `d1p1.img' Finished transfer `d1p1.img' (2.53 MiB/s) Transferring file `d1p2.img' Finished transfer `d1p1.img' (1.48 MiB/s) Transferring file `d1p2.img' Finished transfer `d1p1.img' (770.6 KiB/s) Transferring file `d1p2.img' Finished transfer `d1p2.img' (1.72 MiB/s) Finished transfer `d1p1.img' (1.51 MiB/s) Finished transfer `d1p2.img' (2.49 MiB/s) Transferring file `d1p4.img' Finished transfer `d1p4.img' (127 B/s) Transferring file `d1p5.img' Finished transfer `d1p5.img' (1.95 MiB/s) Finished transfer `d1p2.img' (7.16 MiB/s) Transferring file `d1.fixed_size_partitions' Transferring file `d1.fixed_size_partitions' Finished transfer `d1.fixed_size_partitions' (2 B/s) Transferring file `d1.has_grub' Finished transfer `d1.fixed_size_partitions' (2 B/s) Transferring file `d1.has_grub' Finished transfer `d1.has_grub' Transferring file `d1.mbr' Finished transfer `d1.has_grub' Transferring file `d1.mbr' Transferring file `d1.fixed_size_partitions' Finished transfer `d1.mbr' (821.9 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.fixed_size_partitions' (4 B/s) Transferring file `d1.has_grub' Finished transfer `d1.has_grub' Transferring file `d1.mbr' Finished transfer `d1.minimum.partitions' (184 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.mbr' (892.0 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.original.fstypes' (31 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.minimum.partitions' (184 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.original.swapuuids' (47 B/s) Transferring file `d1.partitions' Finished transfer `d1.partitions' (192 B/s) Transferring file `d1p1.img' Finished transfer `d1.original.fstypes' (33 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.original.swapuuids' Transferring file `d1.partitions' Transferring file `d1.fixed_size_partitions' Finished transfer `d1.partitions' (192 B/s) Transferring file `d1p1.img' Finished transfer `d1.fixed_size_partitions' (1 B/s) Transferring file `d1.mbr' Finished transfer `d1.mbr' (919.7 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.minimum.partitions' (310 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.mbr' (816.3 KiB/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.minimum.partitions' (190 B/s) Transferring file `d1.original.fstypes' Finished transfer `d1.original.fstypes' (80 B/s) Transferring file `d1.original.partitions' Finished transfer `d1.original.fstypes' (30 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.original.swapuuids' Transferring file `d1.partitions' Finished transfer `d1.original.partitions' (310 B/s) Transferring file `d1.original.swapuuids' Finished transfer `d1.partitions' (190 B/s) Transferring file `d1p1.img' Finished transfer `d1.original.swapuuids' (47 B/s) Transferring file `d1p1.img' Finished transfer `d1p1.img' (1.60 MiB/s) Transferring file `d1p2.img' Finished transfer `d1p1.img' (1.84 MiB/s) Transferring file `d1p2.img' Finished transfer `d1p1.img' (1.26 MiB/s) Transferring file `d1p2.img' Finished transfer `d1p1.img' (1.40 MiB/s) Finished transfer `d1p2.img' (1.98 MiB/s) Finished transfer `d1p2.img' (2.83 MiB/s) Transferring file `d1p4.img' Finished transfer `d1p4.img' (127 B/s) Transferring file `d1p5.img' Finished transfer `d1p5.img' (2.24 MiB/s) Finished transfer `d1p2.img' (6.78 MiB/s) Removing old file `d1.original.partitions' Removing old file `d1p4.img' Transferring file `d1.fixed_size_partitions' Finished transfer `d1.fixed_size_partitions' (2 B/s) Transferring file `d1.minimum.partitions' Finished transfer `d1.minimum.partitions' (58 B/s) Removing old file `d1.original.fstypes' Transferring file `d1.original.fstypes' Finished transfer `d1.original.fstypes' (48 B/s) Transferring file `d1.partitions' Finished transfer `d1.partitions' (368 B/s) Transferring file `d1p1.img' Finished transfer `d1p1.img' (9.12 MiB/s) Transferring file `d1p2.img' Finished transfer `d1p2.img' (6.27 MiB/s) Transferring file `d1p4.ebr' Finished transfer `d1p4.ebr' (512 B/s) Transferring file `d1p5.ebr' Finished transfer `d1p5.ebr' (512 B/s) Transferring file `d1p5.img' Finished transfer `d1p5.img' (48.5 KiB/s)
It’s tough to see what’s going on when there are no dates and no full file paths.
This is a bit concerning, also, note the scroll bar…
-
The checksums on the slave node are saying that partitions d1p1, d1p2, and d1p5 were not changed after replication completed.
-
MariaDB [fog]> select * from fileChecksums; +-----------------+----------+------------+------------------------------------------+---------------------------------------------------------------+ | fileChecksumsID | fileHost | fileTime | fileSum | fileLocation | +-----------------+----------+------------+------------------------------------------+---------------------------------------------------------------+ | 1 | 10.0.0.4 | 1456554030 | 7d06f3cc1fc461ef98864ed307742fd1ddf76be1 | /images/postdownloadscripts/fog.postdownload | | 2 | 10.0.0.4 | 1456554030 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/dev/.mntcheck | | 3 | 10.0.0.4 | 1456554030 | e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e | /images/CentOS7Optiplex745UpdateBASE/d1.fixed_size_partitions | | 4 | 10.0.0.4 | 1456554030 | a910a281b36b70a27be8b7752ffe52151bf8b013 | /images/CentOS7Optiplex745UpdateBASE/d1.partitions | | 5 | 10.0.0.4 | 1456554031 | a2d9ccb085562903966212cd08b5b21c63d8754a | /images/CentOS7Optiplex745UpdateBASE/d1.original.fstypes | | 6 | 10.0.0.4 | 1456554031 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.original.swapuuids | | 7 | 10.0.0.4 | 1456554031 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.has_grub | | 8 | 10.0.0.4 | 1456554031 | 4a0e35d60c41d45db95a5168516c9ebea638c41b | /images/CentOS7Optiplex745UpdateBASE/d1.mbr | | 9 | 10.0.0.4 | 1456554031 | de5c8b5deb20cdf3ff4ae685993e6eaff40eb369 | /images/CentOS7Optiplex745UpdateBASE/d1.minimum.partitions | | 10 | 10.0.0.4 | 1456554076 | 21d368ce5f8398bdcccf37f9d9479b6c8d5893b8 | /images/CentOS7Optiplex745UpdateBASE/d1p1.img | | 11 | 10.0.0.4 | 1456554124 | 838381c5d3c2071c55c6c3f57c701c62a02abf0d | /images/CentOS7Optiplex745UpdateBASE/d1p2.img | | 12 | 10.0.0.4 | 1456554125 | 7448d8798a4380162d4b56f9b452e2f6f9e24e7a | /images/Fedora22LivingRoom/d1.fixed_size_partitions | | 13 | 10.0.0.4 | 1456554125 | e383ef7b4700bd13487ec8474389ec62dfb1964f | /images/Fedora22LivingRoom/d1.partitions | | 14 | 10.0.0.4 | 1456554125 | 4fb695c407928af518e7f7820ca5066a5a386c42 | /images/Fedora22LivingRoom/d1.original.fstypes | | 15 | 10.0.0.4 | 1456554125 | 745c70e3928ffbf2a4ac0cd83a49ce641f76d7c5 | /images/Fedora22LivingRoom/d1.original.swapuuids | | 16 | 10.0.0.4 | 1456554125 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora22LivingRoom/d1.has_grub | | 17 | 10.0.0.4 | 1456554125 | 23d9ad9b43a88f0f7159314c8b02ce0aa7db402f | /images/Fedora22LivingRoom/d1.mbr | | 18 | 10.0.0.4 | 1456554125 | 034ab561181f7137919db2ce24c49a3ee4fba7da | /images/Fedora22LivingRoom/d1.minimum.partitions | | 19 | 10.0.0.4 | 1456554230 | 3e89bd52a412e796bb1d47f35ed5efe71d234cb2 | /images/Fedora22LivingRoom/d1p1.img | | 20 | 10.0.0.4 | 1456554231 | ccd1afaabd987136c3be65bd604b6b230c14c92e | /images/Fedora23LivingRoom/d1.fixed_size_partitions | | 21 | 10.0.0.4 | 1456554231 | 861802c73c9400c5de4c81b71fbbcdfb3b131201 | /images/Fedora23LivingRoom/d1.original.partitions | | 22 | 10.0.0.4 | 1456554231 | 5cdab89de5dd88254a298d2f4badd9e4ebf0ee50 | /images/Fedora23LivingRoom/d1.original.fstypes | | 23 | 10.0.0.4 | 1456554231 | a3929b85c1cce6b9934db527550068ccef11f8fd | /images/Fedora23LivingRoom/d1.original.swapuuids | | 24 | 10.0.0.4 | 1456554232 | 7eaa3c92068d8e308ee47728c72adcc35e774a8c | /images/Fedora23LivingRoom/d1.mbr | | 25 | 10.0.0.4 | 1456554232 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora23LivingRoom/d1.has_grub | | 26 | 10.0.0.4 | 1456554232 | 04561e6fb817deacbcbf9bd7cb54d29a1d0cccf0 | /images/Fedora23LivingRoom/d1.minimum.partitions | | 27 | 10.0.0.4 | 1456554236 | ffccd044af5c0295c8fce72c9db39a64b9b96d7a | /images/Fedora23LivingRoom/d1p1.img | | 28 | 10.0.0.4 | 1456554321 | 2496d77b7dc4aa68e5d04b72e5f656461febcc69 | /images/Fedora23LivingRoom/d1p2.img | | 29 | 10.0.0.4 | 1456554322 | 22015e84b062ed609e0f9e84f93d240fa20ff477 | /images/Fedora23LivingRoom/d1p4.img | | 30 | 10.0.0.4 | 1456554323 | 98bfd22dbf59c8ba3398a2d7c542f265768e9675 | /images/Fedora23LivingRoom/d1p5.img | | 31 | 10.0.0.4 | 1456554323 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/.mntcheck | | 32 | 10.0.0.4 | 1456554323 | adc83b19e793491b1c6ea0fd8b46cd9f32e592fc | /images/Win7/d1.fixed_size_partitions | | 33 | 10.0.0.4 | 1456554324 | 3ed69cf742c34690cc061eb44d34482e87b90776 | /images/Win7/d1.partitions | | 34 | 10.0.0.4 | 1456554324 | d9bdc2da3027044ae124831e8cca1b2f534d7163 | /images/Win7/d1.original.fstypes | | 35 | 10.0.0.4 | 1456554324 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Win7/d1.original.swapuuids | | 36 | 10.0.0.4 | 1456554324 | 4a9712192b5b0c272cf0797fe033ab2f7956d79d | /images/Win7/d1.mbr | | 37 | 10.0.0.4 | 1456554324 | 139c5dbc9b92a04f0966c90d4fd5c1ff8a4a9eb1 | /images/Win7/d1.minimum.partitions | | 38 | 10.0.0.4 | 1456554324 | d3e3953115255bdd8cedb03a6ada3a5a69f9c3b3 | /images/Win7/d1p1.img | | 39 | 10.0.0.4 | 1456554652 | a7f07cef8abbdaf0137cb8e1191e22e33f75d863 | /images/Win7/d1p2.img | | 40 | 10.0.0.4 | 1456554652 | f09f7644b27f9b6964c2f35028dfc406bdad22b6 | /opt/fog/snapins/ssl/.srvprivate.key | | 41 | 10.0.0.4 | 1456554652 | 606a535989dfa30f585b0b2b96ff3e47d9df4dc7 | /opt/fog/snapins/ssl/fog.csr | | 42 | 10.0.0.4 | 1456554653 | d8dcc09a122213729f2de9937ea77d984ea01323 | /opt/fog/snapins/ssl/CA/.fogCA.key | | 43 | 10.0.0.4 | 1456554653 | b0564bc10ecbedc66e61766c27bfdfacf7c508fa | /opt/fog/snapins/ssl/CA/.fogCA.pem | | 44 | 10.0.0.4 | 1456554653 | 0a4b6296e1f413526bca330f50ed4661a2b5d381 | /opt/fog/snapins/ssl/CA/.srl | | 45 | 10.0.0.4 | 1456554654 | d92e5813b41b1e24c81dcc8ac9efa66f27472c70 | /opt/fog/snapins/googlechromestandaloneenterprise.msi | | 46 | 10.0.0.5 | 1456554727 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/.mntcheck | | 47 | 10.0.0.5 | 1456554727 | 7d06f3cc1fc461ef98864ed307742fd1ddf76be1 | /images/postdownloadscripts/fog.postdownload | | 48 | 10.0.0.5 | 1456554727 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/dev/.mntcheck | | 49 | 10.0.0.5 | 1456554727 | e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e | /images/CentOS7Optiplex745UpdateBASE/d1.fixed_size_partitions | | 50 | 10.0.0.5 | 1456554728 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.has_grub | | 51 | 10.0.0.5 | 1456554728 | 4a0e35d60c41d45db95a5168516c9ebea638c41b | /images/CentOS7Optiplex745UpdateBASE/d1.mbr | | 52 | 10.0.0.5 | 1456554728 | de5c8b5deb20cdf3ff4ae685993e6eaff40eb369 | /images/CentOS7Optiplex745UpdateBASE/d1.minimum.partitions | | 53 | 10.0.0.5 | 1456554728 | a2d9ccb085562903966212cd08b5b21c63d8754a | /images/CentOS7Optiplex745UpdateBASE/d1.original.fstypes | | 54 | 10.0.0.5 | 1456554728 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.original.swapuuids | | 55 | 10.0.0.5 | 1456554728 | a910a281b36b70a27be8b7752ffe52151bf8b013 | /images/CentOS7Optiplex745UpdateBASE/d1.partitions | | 56 | 10.0.0.5 | 1456554763 | 21d368ce5f8398bdcccf37f9d9479b6c8d5893b8 | /images/CentOS7Optiplex745UpdateBASE/d1p1.img | | 57 | 10.0.0.5 | 1456554831 | 838381c5d3c2071c55c6c3f57c701c62a02abf0d | /images/CentOS7Optiplex745UpdateBASE/d1p2.img | | 58 | 10.0.0.5 | 1456554833 | 7448d8798a4380162d4b56f9b452e2f6f9e24e7a | /images/Fedora22LivingRoom/d1.fixed_size_partitions | | 59 | 10.0.0.5 | 1456554833 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora22LivingRoom/d1.has_grub | | 60 | 10.0.0.5 | 1456554833 | 23d9ad9b43a88f0f7159314c8b02ce0aa7db402f | /images/Fedora22LivingRoom/d1.mbr | | 61 | 10.0.0.5 | 1456554833 | 034ab561181f7137919db2ce24c49a3ee4fba7da | /images/Fedora22LivingRoom/d1.minimum.partitions | | 62 | 10.0.0.5 | 1456554833 | 4fb695c407928af518e7f7820ca5066a5a386c42 | /images/Fedora22LivingRoom/d1.original.fstypes | | 63 | 10.0.0.5 | 1456554833 | 745c70e3928ffbf2a4ac0cd83a49ce641f76d7c5 | /images/Fedora22LivingRoom/d1.original.swapuuids | | 64 | 10.0.0.5 | 1456554833 | e383ef7b4700bd13487ec8474389ec62dfb1964f | /images/Fedora22LivingRoom/d1.partitions | | 65 | 10.0.0.5 | 1456554939 | 3e89bd52a412e796bb1d47f35ed5efe71d234cb2 | /images/Fedora22LivingRoom/d1p1.img | | 66 | 10.0.0.5 | 1456554940 | ccd1afaabd987136c3be65bd604b6b230c14c92e | /images/Fedora23LivingRoom/d1.fixed_size_partitions | | 67 | 10.0.0.5 | 1456554940 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora23LivingRoom/d1.has_grub | | 68 | 10.0.0.5 | 1456554940 | 7eaa3c92068d8e308ee47728c72adcc35e774a8c | /images/Fedora23LivingRoom/d1.mbr | | 69 | 10.0.0.5 | 1456554940 | 04561e6fb817deacbcbf9bd7cb54d29a1d0cccf0 | /images/Fedora23LivingRoom/d1.minimum.partitions | | 70 | 10.0.0.5 | 1456554940 | 5cdab89de5dd88254a298d2f4badd9e4ebf0ee50 | /images/Fedora23LivingRoom/d1.original.fstypes | | 71 | 10.0.0.5 | 1456554940 | 861802c73c9400c5de4c81b71fbbcdfb3b131201 | /images/Fedora23LivingRoom/d1.original.partitions | | 72 | 10.0.0.5 | 1456554940 | a3929b85c1cce6b9934db527550068ccef11f8fd | /images/Fedora23LivingRoom/d1.original.swapuuids | | 73 | 10.0.0.5 | 1456554945 | ffccd044af5c0295c8fce72c9db39a64b9b96d7a | /images/Fedora23LivingRoom/d1p1.img | | 74 | 10.0.0.5 | 1456555023 | 2496d77b7dc4aa68e5d04b72e5f656461febcc69 | /images/Fedora23LivingRoom/d1p2.img | | 75 | 10.0.0.5 | 1456555024 | 22015e84b062ed609e0f9e84f93d240fa20ff477 | /images/Fedora23LivingRoom/d1p4.img | | 76 | 10.0.0.5 | 1456555025 | 98bfd22dbf59c8ba3398a2d7c542f265768e9675 | /images/Fedora23LivingRoom/d1p5.img | | 77 | 10.0.0.5 | 1456555025 | adc83b19e793491b1c6ea0fd8b46cd9f32e592fc | /images/Win7/d1.fixed_size_partitions | | 78 | 10.0.0.5 | 1456555025 | 4a9712192b5b0c272cf0797fe033ab2f7956d79d | /images/Win7/d1.mbr | | 79 | 10.0.0.5 | 1456555026 | 139c5dbc9b92a04f0966c90d4fd5c1ff8a4a9eb1 | /images/Win7/d1.minimum.partitions | | 80 | 10.0.0.5 | 1456555026 | d9bdc2da3027044ae124831e8cca1b2f534d7163 | /images/Win7/d1.original.fstypes | | 81 | 10.0.0.5 | 1456555026 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Win7/d1.original.swapuuids | | 82 | 10.0.0.5 | 1456555026 | 3ed69cf742c34690cc061eb44d34482e87b90776 | /images/Win7/d1.partitions | | 83 | 10.0.0.5 | 1456555026 | d3e3953115255bdd8cedb03a6ada3a5a69f9c3b3 | /images/Win7/d1p1.img | | 84 | 10.0.0.5 | 1456555263 | a7f07cef8abbdaf0137cb8e1191e22e33f75d863 | /images/Win7/d1p2.img | | 85 | 10.0.0.5 | 1456555264 | f09f7644b27f9b6964c2f35028dfc406bdad22b6 | /opt/fog/snapins/ssl/.srvprivate.key | | 86 | 10.0.0.5 | 1456555264 | 606a535989dfa30f585b0b2b96ff3e47d9df4dc7 | /opt/fog/snapins/ssl/fog.csr | | 87 | 10.0.0.5 | 1456555264 | d8dcc09a122213729f2de9937ea77d984ea01323 | /opt/fog/snapins/ssl/CA/.fogCA.key | | 88 | 10.0.0.5 | 1456555264 | b0564bc10ecbedc66e61766c27bfdfacf7c508fa | /opt/fog/snapins/ssl/CA/.fogCA.pem | | 89 | 10.0.0.5 | 1456555264 | 15c32736d8e14df26be33c8d12e0dc0539d43822 | /opt/fog/snapins/ssl/CA/.srl | | 90 | 10.0.0.4 | 1456596538 | 7d06f3cc1fc461ef98864ed307742fd1ddf76be1 | /images/postdownloadscripts/fog.postdownload | | 91 | 10.0.0.4 | 1456596538 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/dev/.mntcheck | | 92 | 10.0.0.4 | 1456596538 | e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e | /images/CentOS7Optiplex745UpdateBASE/d1.fixed_size_partitions | | 93 | 10.0.0.4 | 1456596538 | a910a281b36b70a27be8b7752ffe52151bf8b013 | /images/CentOS7Optiplex745UpdateBASE/d1.partitions | | 94 | 10.0.0.4 | 1456596538 | a2d9ccb085562903966212cd08b5b21c63d8754a | /images/CentOS7Optiplex745UpdateBASE/d1.original.fstypes | | 95 | 10.0.0.4 | 1456596538 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.original.swapuuids | | 96 | 10.0.0.4 | 1456596538 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.has_grub | | 97 | 10.0.0.4 | 1456596538 | 4a0e35d60c41d45db95a5168516c9ebea638c41b | /images/CentOS7Optiplex745UpdateBASE/d1.mbr | | 98 | 10.0.0.4 | 1456596538 | de5c8b5deb20cdf3ff4ae685993e6eaff40eb369 | /images/CentOS7Optiplex745UpdateBASE/d1.minimum.partitions | | 99 | 10.0.0.4 | 1456596568 | 21d368ce5f8398bdcccf37f9d9479b6c8d5893b8 | /images/CentOS7Optiplex745UpdateBASE/d1p1.img | | 100 | 10.0.0.4 | 1456596612 | 838381c5d3c2071c55c6c3f57c701c62a02abf0d | /images/CentOS7Optiplex745UpdateBASE/d1p2.img | | 101 | 10.0.0.4 | 1456596613 | 7448d8798a4380162d4b56f9b452e2f6f9e24e7a | /images/Fedora22LivingRoom/d1.fixed_size_partitions | | 102 | 10.0.0.4 | 1456596613 | e383ef7b4700bd13487ec8474389ec62dfb1964f | /images/Fedora22LivingRoom/d1.partitions | | 103 | 10.0.0.4 | 1456596613 | 4fb695c407928af518e7f7820ca5066a5a386c42 | /images/Fedora22LivingRoom/d1.original.fstypes | | 104 | 10.0.0.4 | 1456596613 | 745c70e3928ffbf2a4ac0cd83a49ce641f76d7c5 | /images/Fedora22LivingRoom/d1.original.swapuuids | | 105 | 10.0.0.4 | 1456596613 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora22LivingRoom/d1.has_grub | | 106 | 10.0.0.4 | 1456596614 | 23d9ad9b43a88f0f7159314c8b02ce0aa7db402f | /images/Fedora22LivingRoom/d1.mbr | | 107 | 10.0.0.4 | 1456596614 | 034ab561181f7137919db2ce24c49a3ee4fba7da | /images/Fedora22LivingRoom/d1.minimum.partitions | | 108 | 10.0.0.4 | 1456596681 | 3e89bd52a412e796bb1d47f35ed5efe71d234cb2 | /images/Fedora22LivingRoom/d1p1.img | | 109 | 10.0.0.4 | 1456596682 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/.mntcheck | | 110 | 10.0.0.4 | 1456596682 | adc83b19e793491b1c6ea0fd8b46cd9f32e592fc | /images/Win7/d1.fixed_size_partitions | | 111 | 10.0.0.4 | 1456596682 | 3ed69cf742c34690cc061eb44d34482e87b90776 | /images/Win7/d1.partitions | | 112 | 10.0.0.4 | 1456596682 | d9bdc2da3027044ae124831e8cca1b2f534d7163 | /images/Win7/d1.original.fstypes | | 113 | 10.0.0.4 | 1456596682 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Win7/d1.original.swapuuids | | 114 | 10.0.0.4 | 1456596682 | 4a9712192b5b0c272cf0797fe033ab2f7956d79d | /images/Win7/d1.mbr | | 115 | 10.0.0.4 | 1456596682 | 139c5dbc9b92a04f0966c90d4fd5c1ff8a4a9eb1 | /images/Win7/d1.minimum.partitions | | 116 | 10.0.0.4 | 1456596683 | d3e3953115255bdd8cedb03a6ada3a5a69f9c3b3 | /images/Win7/d1p1.img | | 117 | 10.0.0.4 | 1456597082 | a7f07cef8abbdaf0137cb8e1191e22e33f75d863 | /images/Win7/d1p2.img | | 118 | 10.0.0.4 | 1456597086 | b40761a713f76d25e4d002eaf84ef2c5d1a79751 | /images/Fedora23LivingRoom/d1.fixed_size_partitions | | 119 | 10.0.0.4 | 1456597087 | 950c35c0634b8c6eef5c8ffac48112b5557a0ef6 | /images/Fedora23LivingRoom/d1.partitions | | 120 | 10.0.0.4 | 1456597087 | 74453471faea8f6ba64880b066e79cc5ad175693 | /images/Fedora23LivingRoom/d1.original.fstypes | | 121 | 10.0.0.4 | 1456597087 | 337d3942eb95bf3acbbb8f445752183f3f484e0e | /images/Fedora23LivingRoom/d1.original.swapuuids | | 122 | 10.0.0.4 | 1456597087 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora23LivingRoom/d1.has_grub | | 123 | 10.0.0.4 | 1456597087 | 68e325611ffaf97a8256c00158ed8c5ebb58f50d | /images/Fedora23LivingRoom/d1.mbr | | 124 | 10.0.0.4 | 1456597087 | 151ddf4da950213ec9ba8cf22c0bad21e11e83d4 | /images/Fedora23LivingRoom/d1.minimum.partitions | | 125 | 10.0.0.4 | 1456597087 | 5dc2ff987315fb208d54ac2ce473af7f46e77533 | /images/Fedora23LivingRoom/d1p4.ebr | | 126 | 10.0.0.4 | 1456597087 | 5c3eb80066420002bc3dcc7ca4ab6efad7ed4ae5 | /images/Fedora23LivingRoom/d1p5.ebr | | 127 | 10.0.0.4 | 1456597092 | 22f68eb0f40c50279048121f2e521099152f0d29 | /images/Fedora23LivingRoom/d1p1.img | | 128 | 10.0.0.4 | 1456597232 | 6b671edc75587fa51750c5f1d8c71e7383df0c87 | /images/Fedora23LivingRoom/d1p2.img | | 129 | 10.0.0.4 | 1456597234 | 06672a79cfdf1b66552f33d306f21cd286a45567 | /images/Fedora23LivingRoom/d1p5.img | | 130 | 10.0.0.4 | 1456597235 | f09f7644b27f9b6964c2f35028dfc406bdad22b6 | /opt/fog/snapins/ssl/.srvprivate.key | | 131 | 10.0.0.4 | 1456597235 | 606a535989dfa30f585b0b2b96ff3e47d9df4dc7 | /opt/fog/snapins/ssl/fog.csr | | 132 | 10.0.0.4 | 1456597235 | d8dcc09a122213729f2de9937ea77d984ea01323 | /opt/fog/snapins/ssl/CA/.fogCA.key | | 133 | 10.0.0.4 | 1456597235 | b0564bc10ecbedc66e61766c27bfdfacf7c508fa | /opt/fog/snapins/ssl/CA/.fogCA.pem | | 134 | 10.0.0.4 | 1456597235 | c6d47b64b1a6ff598a84ca596a894375956d4371 | /opt/fog/snapins/ssl/CA/.srl | | 135 | 10.0.0.4 | 1456597237 | d92e5813b41b1e24c81dcc8ac9efa66f27472c70 | /opt/fog/snapins/googlechromestandaloneenterprise.msi | | 136 | 10.0.0.5 | 1456597523 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/.mntcheck | | 137 | 10.0.0.5 | 1456597523 | 7d06f3cc1fc461ef98864ed307742fd1ddf76be1 | /images/postdownloadscripts/fog.postdownload | | 138 | 10.0.0.5 | 1456597523 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/dev/.mntcheck | | 139 | 10.0.0.5 | 1456597523 | e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e | /images/CentOS7Optiplex745UpdateBASE/d1.fixed_size_partitions | | 140 | 10.0.0.5 | 1456597523 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.has_grub | | 141 | 10.0.0.5 | 1456597523 | 4a0e35d60c41d45db95a5168516c9ebea638c41b | /images/CentOS7Optiplex745UpdateBASE/d1.mbr | | 142 | 10.0.0.5 | 1456597523 | de5c8b5deb20cdf3ff4ae685993e6eaff40eb369 | /images/CentOS7Optiplex745UpdateBASE/d1.minimum.partitions | | 143 | 10.0.0.5 | 1456597523 | a2d9ccb085562903966212cd08b5b21c63d8754a | /images/CentOS7Optiplex745UpdateBASE/d1.original.fstypes | | 144 | 10.0.0.5 | 1456597523 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/CentOS7Optiplex745UpdateBASE/d1.original.swapuuids | | 145 | 10.0.0.5 | 1456597523 | a910a281b36b70a27be8b7752ffe52151bf8b013 | /images/CentOS7Optiplex745UpdateBASE/d1.partitions | | 146 | 10.0.0.5 | 1456597556 | 21d368ce5f8398bdcccf37f9d9479b6c8d5893b8 | /images/CentOS7Optiplex745UpdateBASE/d1p1.img | | 147 | 10.0.0.5 | 1456597605 | 838381c5d3c2071c55c6c3f57c701c62a02abf0d | /images/CentOS7Optiplex745UpdateBASE/d1p2.img | | 148 | 10.0.0.5 | 1456597606 | 7448d8798a4380162d4b56f9b452e2f6f9e24e7a | /images/Fedora22LivingRoom/d1.fixed_size_partitions | | 149 | 10.0.0.5 | 1456597606 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora22LivingRoom/d1.has_grub | | 150 | 10.0.0.5 | 1456597606 | 23d9ad9b43a88f0f7159314c8b02ce0aa7db402f | /images/Fedora22LivingRoom/d1.mbr | | 151 | 10.0.0.5 | 1456597606 | 034ab561181f7137919db2ce24c49a3ee4fba7da | /images/Fedora22LivingRoom/d1.minimum.partitions | | 152 | 10.0.0.5 | 1456597606 | 4fb695c407928af518e7f7820ca5066a5a386c42 | /images/Fedora22LivingRoom/d1.original.fstypes | | 153 | 10.0.0.5 | 1456597606 | 745c70e3928ffbf2a4ac0cd83a49ce641f76d7c5 | /images/Fedora22LivingRoom/d1.original.swapuuids | | 154 | 10.0.0.5 | 1456597606 | e383ef7b4700bd13487ec8474389ec62dfb1964f | /images/Fedora22LivingRoom/d1.partitions | | 155 | 10.0.0.5 | 1456597685 | 3e89bd52a412e796bb1d47f35ed5efe71d234cb2 | /images/Fedora22LivingRoom/d1p1.img | | 156 | 10.0.0.5 | 1456597686 | bd0125f265897b42f3a9c1cd065c26ac4d486443 | /images/Fedora23LivingRoom/d1.fixed_size_partitions | | 157 | 10.0.0.5 | 1456597686 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Fedora23LivingRoom/d1.has_grub | | 158 | 10.0.0.5 | 1456597686 | 7eaa3c92068d8e308ee47728c72adcc35e774a8c | /images/Fedora23LivingRoom/d1.mbr | | 159 | 10.0.0.5 | 1456597686 | 0af0c2e070c0a318a5167484cb18f55703e49528 | /images/Fedora23LivingRoom/d1.minimum.partitions | | 160 | 10.0.0.5 | 1456597686 | a3929b85c1cce6b9934db527550068ccef11f8fd | /images/Fedora23LivingRoom/d1.original.swapuuids | | 161 | 10.0.0.5 | 1456597690 | 5cfd63fe1107d898c6588fff459e6dba0ecb6cce | /images/Fedora23LivingRoom/d1p1.img | | 162 | 10.0.0.5 | 1456597797 | a4f5da5548d94ef13c260ce7069b1ce24ec43fed | /images/Fedora23LivingRoom/d1p2.img | | 163 | 10.0.0.5 | 1456597798 | 62449efcb9b1142a77097f652b2f6148ecb9e70f | /images/Fedora23LivingRoom/d1p5.img | | 164 | 10.0.0.5 | 1456597798 | 74453471faea8f6ba64880b066e79cc5ad175693 | /images/Fedora23LivingRoom/d1.original.fstypes | | 165 | 10.0.0.5 | 1456597799 | 950c35c0634b8c6eef5c8ffac48112b5557a0ef6 | /images/Fedora23LivingRoom/d1.partitions | | 166 | 10.0.0.5 | 1456597799 | 5dc2ff987315fb208d54ac2ce473af7f46e77533 | /images/Fedora23LivingRoom/d1p4.ebr | | 167 | 10.0.0.5 | 1456597799 | 5c3eb80066420002bc3dcc7ca4ab6efad7ed4ae5 | /images/Fedora23LivingRoom/d1p5.ebr | | 168 | 10.0.0.5 | 1456597799 | adc83b19e793491b1c6ea0fd8b46cd9f32e592fc | /images/Win7/d1.fixed_size_partitions | | 169 | 10.0.0.5 | 1456597799 | 4a9712192b5b0c272cf0797fe033ab2f7956d79d | /images/Win7/d1.mbr | | 170 | 10.0.0.5 | 1456597799 | 139c5dbc9b92a04f0966c90d4fd5c1ff8a4a9eb1 | /images/Win7/d1.minimum.partitions | | 171 | 10.0.0.5 | 1456597799 | d9bdc2da3027044ae124831e8cca1b2f534d7163 | /images/Win7/d1.original.fstypes | | 172 | 10.0.0.5 | 1456597799 | da39a3ee5e6b4b0d3255bfef95601890afd80709 | /images/Win7/d1.original.swapuuids | | 173 | 10.0.0.5 | 1456597799 | 3ed69cf742c34690cc061eb44d34482e87b90776 | /images/Win7/d1.partitions | | 174 | 10.0.0.5 | 1456597800 | d3e3953115255bdd8cedb03a6ada3a5a69f9c3b3 | /images/Win7/d1p1.img | | 175 | 10.0.0.5 | 1456598047 | a7f07cef8abbdaf0137cb8e1191e22e33f75d863 | /images/Win7/d1p2.img | | 176 | 10.0.0.5 | 1456598048 | f09f7644b27f9b6964c2f35028dfc406bdad22b6 | /opt/fog/snapins/ssl/.srvprivate.key | | 177 | 10.0.0.5 | 1456598048 | 606a535989dfa30f585b0b2b96ff3e47d9df4dc7 | /opt/fog/snapins/ssl/fog.csr | | 178 | 10.0.0.5 | 1456598048 | d8dcc09a122213729f2de9937ea77d984ea01323 | /opt/fog/snapins/ssl/CA/.fogCA.key | | 179 | 10.0.0.5 | 1456598048 | b0564bc10ecbedc66e61766c27bfdfacf7c508fa | /opt/fog/snapins/ssl/CA/.fogCA.pem | | 180 | 10.0.0.5 | 1456598048 | 0a4b6296e1f413526bca330f50ed4661a2b5d381 | /opt/fog/snapins/ssl/CA/.srl | +-----------------+----------+------------+------------------------------------------+---------------------------------------------------------------+ 180 rows in set (0.01 sec)
-
Look at these particular checksum records from the master and slave, taken before the image was uploaded, and after upload and after replication:
27 | 10.0.0.4 | 1456554236 | ffccd044af5c0295c8fce72c9db39a64b9b96d7a | /images/Fedora23LivingRoom/d1p1.img 73 | 10.0.0.5 | 1456554945 | ffccd044af5c0295c8fce72c9db39a64b9b96d7a | /images/Fedora23LivingRoom/d1p1.img then capture a new image, wait for replication.... New checksums: 127 | 10.0.0.4 | 1456597092 | 22f68eb0f40c50279048121f2e521099152f0d29 | /images/Fedora23LivingRoom/d1p1.img 161 | 10.0.0.5 | 1456597690 | 5cfd63fe1107d898c6588fff459e6dba0ecb6cce | /images/Fedora23LivingRoom/d1p1.img
We can see that yes, these files were absolutely identical prior to the image being updated, on both nodes in the group.
After the image was updated, and after replication happened, we see that the files were changed, but are not identical. -
If you look at my checksum records for my Win7 image from this thread, you’ll see that the same problem existed then too:
https://forums.fogproject.org/topic/6318/image-file-integrity/12Only then, I didn’t “try out” my image so I was unaware of an issue and thought something was wrong with the logic I wrote, but I don’t think anything is wrong with it now. Looking at the actual data from both those tests and now, It’s clear there is a replication issue.
-
Yesterday evening, I was able to work with Tom in his test-bed to replicate the problem. Turns out, it’s a bug in
lftp
itself. Here’s the bug report that Tom found:
https://github.com/lavv17/lftp/issues/148Fedora 23 ships with lftp version 4.6.1 so it has the bug. If the file being evaluated is smaller, it gets replaced correctly. If the file being evaluated is larger, lftp only adds the changed bits - resulting in a mostly old image that is highly corrupted.
Tom said he would work on using something else for replication in the mean time.
-
I added some preliminary checking to try to figure out if a file is the same. If it isn’t, it deletes it on the remote side, and then continues checking the other files. This means lftp should always start a fresh file.