FOG nodes not in sync
-
du -m --max-depth=1 /images
Park City Master Node:
3198 /images/TulsaPBX 11681 /images/DellOptiplex7050Win10Pro3-1-18 18792 /images/DellPrec5530Win10ProUEFI9-27-18 10986 /images/DellOptiplex7040Win10Pro3-5-18 2386 /images/DenverPBX14 52357 /images/DellLatitude7280Win10Pro3-1-18 16665 /images/DellLatitudeE55709-12-18 8675 /images/DellLatitudeE6420Win7Pro3-20-18 24192 /images/LAPBX12Backup 4868 /images/OrlandoFreePBX12Backup 21612 /images/DellLatitudeE7250Win10Pro3-13-18 17532 /images/DellLatitude7290Win10Pro8-3-18 2770 /images/ClevelandFreePBX13Backup 956 /images/FreePBX12FreshInstall 6514 /images/RaleighPBX12 1730 /images/FresnoPBX14 11903 /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI 12237 /images/LenovoX240Win10Pro4-9-18 1679 /images/FreePBX13Generic 5338 /images/SuperMicroServer2012R2forNAS 3192 /images/Windows7ProGeneric64-bit 954 /images/FreshFreePBX12 1226 /images/FreshFreePBX13 2342 /images/SeattlePBX14 13003 /images/DellLatitude7290GenericTest 8444 /images/SuperMicroWin10ProVol1709forNAS 18261 /images/DellLatitude7490Win10Pro8-1-18 1259 /images/St.LouisPBX-12 1511 /images/FreePBX14Generic 3201 /images/bigsnapins 6650 /images/BostonPBX12backup 12387 /images/DellLatitude5290Win10Pro3-16-18 12979 /images/LenovoX230Updated 2594 /images/FreePBX14-1805-3 13403 /images/DellLatitudeE7270Win10Pro3-1-18 24049 /images/DellLatitude7280Win10Pro3-20-18 2520 /images/OrlandoFreePBX14 14793 /images/DellLatitude5480Win10Pro3-1-18 2501 /images/LAPBX14 2426 /images/RaleighPBX14 2645 /images/SeattlePBX12backup 3562 /images/DetroitPBXBackup13 4224 /images/AtlantaPBX14 1 /images/postdownloadscripts 2512 /images/BostonPBX14 19018 /images/DellAlienwareBaseWin10Pro9-21-18 26821 /images/DellLatitude5480Win10Pro6-4-18 31185 /images/Win-Pak 3980 /images/Win10ProGeneric 2584 /images/St.LouisPBX-14 3256 /images/13to14PBXupgradetest 5874 /images/Windows10Pro64bit1709Vol 10914 /images/LenovoX240Win7Pro3-28-18 2673 /images/dev 1948 /images/FreePBX13-Ext4-Base 1 /images/lost+found 18475 /images/DellLatitudeE5470Win10Pro3-8-18 7164 /images/Windows10Pro64bit1803Vol 25542 /images/DellLatitudeE5450Win10Pro3-1-18 4452 /images/WindowsServer2016Generic 2824 /images/PortlandPBX14 3436 /images/PortlandPBX12Backup 28037 /images/DellAlienwareDevelopersWin10Pro9-27-18 16440 /images/DellOptiplex9020Win10Pro5-25-18 2495 /images/DaytonPBX14 4695 /images/DaytonPBX12 618490 /imagesPark City Storage Node:
3256 /images/13to14PBXupgradetest 11587 /images/DellLatitude7280Win10Pro3-20-18 8675 /images/DellLatitudeE6420Win7Pro3-20-18 1 /images/postdownloadscripts 9491 /images/DellOptiplex7050Win10Pro3-1-18 4452 /images/WindowsServer2016Generic 3980 /images/Win10ProGeneric 8444 /images/SuperMicroWin10ProVol1709forNAS 2485 /images/DaytonPBX14 1511 /images/FreePBX14Generic 1948 /images/FreePBX13-Ext4-Base 12237 /images/LenovoX240Win10Pro4-9-18 20576 /images/DellOptiplex9020Win10Pro5-25-18 4695 /images/DaytonPBX12 2645 /images/SeattlePBX12backup 1831 /images/FreePBX14-1805-3 5874 /images/Windows10Pro64bit1709Vol 12420 /images/DellLatitudeE5450Win10Pro3-1-18 956 /images/FreePBX12FreshInstall 2520 /images/OrlandoFreePBX14 31185 /images/Win-Pak 11903 /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI 11648 /images/DellLatitudeE7270Win10Pro3-1-18 5039 /images/AtlantaPBX14 2349 /images/RaleighPBX14 6650 /images/BostonPBX12backup 5338 /images/SuperMicroServer2012R2forNAS 3192 /images/Windows7ProGeneric64-bit 2770 /images/ClevelandFreePBX13Backup 3436 /images/PortlandPBX12Backup 2512 /images/BostonPBX14 3578 /images/DetroitPBXBackup13 11500 /images/DellOptiplex7040Win10Pro3-5-18 14565 /images/DellLatitudeE5470Win10Pro3-8-18 3509 /images/St.LouisPBX-14 2342 /images/SeattlePBX14 2501 /images/LAPBX14 1259 /images/St.LouisPBX-12 1 /images/dev 11372 /images/DellLatitudeE7250Win10Pro3-13-18 10914 /images/LenovoX240Win7Pro3-28-18 4868 /images/OrlandoFreePBX12Backup 6514 /images/RaleighPBX12 2573 /images/DenverPBX14 14793 /images/DellLatitude5480Win10Pro3-1-18 2824 /images/PortlandPBX14 298697 /imagesBuffalo Storage Node:
12237 /images/LenovoX240Win10Pro4-9-18 956 /images/FreePBX12FreshInstall 4695 /images/DaytonPBX12 52357 /images/DellLatitude7280Win10Pro3-1-18 11681 /images/DellOptiplex7050Win10Pro3-1-18 2594 /images/FreePBX14-1805-3 2520 /images/OrlandoFreePBX14 2645 /images/SeattlePBX12backup 1226 /images/FreshFreePBX13 3562 /images/DetroitPBXBackup13 24192 /images/LAPBX12Backup 2584 /images/St.LouisPBX-14 1259 /images/St.LouisPBX-12 1730 /images/FresnoPBX14 10914 /images/LenovoX240Win7Pro3-28-18 5874 /images/Windows10Pro64bit1709Vol 10986 /images/DellOptiplex7040Win10Pro3-5-18 5338 /images/SuperMicroServer2012R2forNAS 2342 /images/SeattlePBX14 18261 /images/DellLatitude7490Win10Pro8-1-18 2386 /images/DenverPBX14 4868 /images/OrlandoFreePBX12Backup 16665 /images/DellLatitudeE55709-12-18 3198 /images/TulsaPBX 12979 /images/LenovoX230Updated 13403 /images/DellLatitudeE7270Win10Pro3-1-18 1679 /images/FreePBX13Generic 1 /images/bigsnapins 3436 /images/PortlandPBX12Backup 3192 /images/Windows7ProGeneric64-bit 14793 /images/DellLatitude5480Win10Pro3-1-18 6514 /images/RaleighPBX12 16440 /images/DellOptiplex9020Win10Pro5-25-18 2495 /images/DaytonPBX14 7607 /images/WindowsServer2016Generic 25542 /images/DellLatitudeE5450Win10Pro3-1-18 1 /images/dev 18475 /images/DellLatitudeE5470Win10Pro3-8-18 8675 /images/DellLatitudeE6420Win7Pro3-20-18 19018 /images/DellAlienwareBaseWin10Pro9-21-18 1 /images/lost+found 1511 /images/FreePBX14Generic 17532 /images/DellLatitude7290Win10Pro8-3-18 12387 /images/DellLatitude5290Win10Pro3-16-18 7164 /images/Windows10Pro64bit1803Vol 4224 /images/AtlantaPBX14 3980 /images/Win10ProGeneric 2512 /images/BostonPBX14 8444 /images/SuperMicroWin10ProVol1709forNAS 1948 /images/FreePBX13-Ext4-Base 2501 /images/LAPBX14 11903 /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI 6650 /images/BostonPBX12backup 21612 /images/DellLatitudeE7250Win10Pro3-13-18 2770 /images/ClevelandFreePBX13Backup 2426 /images/RaleighPBX14 954 /images/FreshFreePBX12 26821 /images/DellLatitude5480Win10Pro6-4-18 31185 /images/Win-Pak 2824 /images/PortlandPBX14 24049 /images/DellLatitude7280Win10Pro3-20-18 18792 /images/DellPrec5530Win10ProUEFI9-27-18 3256 /images/13to14PBXupgradetest 13003 /images/DellLatitude7290GenericTest 28037 /images/DellAlienwareDevelopersWin10Pro9-27-18 1 /images/postdownloadscripts 615772 /imagesDallas Storage Node:
12387 /images/DellLatitude5290Win10Pro3-16-18 1 /images/postdownloadscripts 14565 /images/DellLatitudeE5470Win10Pro3-8-18 12237 /images/LenovoX240Win10Pro4-9-18 19018 /images/DellAlienwareBaseWin10Pro9-21-18 1948 /images/FreePBX13-Ext4-Base 11372 /images/DellLatitudeE7250Win10Pro3-13-18 6514 /images/RaleighPBX12 5338 /images/SuperMicroServer2012R2forNAS 956 /images/FreePBX12FreshInstall 2594 /images/FreePBX14-1805-3 2501 /images/LAPBX14 3980 /images/Win10ProGeneric 20576 /images/DellOptiplex9020Win10Pro5-25-18 8444 /images/SuperMicroWin10ProVol1709forNAS 2386 /images/DenverPBX14 4868 /images/OrlandoFreePBX12Backup 24495 /images/DellLatitudeE5450Win10Pro3-1-18 11903 /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI 11587 /images/DellLatitude7280Win10Pro3-20-18 6650 /images/BostonPBX12backup 31185 /images/Win-Pak 21268 /images/DellLatitude7280Win10Pro3-1-18 5874 /images/Windows10Pro64bit1709Vol 28037 /images/DellAlienwareDevelopersWin10Pro9-27-18 9491 /images/DellOptiplex7050Win10Pro3-1-18 2584 /images/St.LouisPBX-14 2512 /images/BostonPBX14 11500 /images/DellOptiplex7040Win10Pro3-5-18 8675 /images/DellLatitudeE6420Win7Pro3-20-18 2520 /images/OrlandoFreePBX14 2342 /images/SeattlePBX14 4452 /images/WindowsServer2016Generic 14793 /images/DellLatitude5480Win10Pro3-1-18 3198 /images/TulsaPBX 3436 /images/PortlandPBX12Backup 11648 /images/DellLatitudeE7270Win10Pro3-1-18 2495 /images/DaytonPBX14 3192 /images/Windows7ProGeneric64-bit 1 /images/dev 2824 /images/PortlandPBX14 7164 /images/Windows10Pro64bit1803Vol 3256 /images/13to14PBXupgradetest 2770 /images/ClevelandFreePBX13Backup 4695 /images/DaytonPBX12 1259 /images/St.LouisPBX-12 3578 /images/DetroitPBXBackup13 2426 /images/RaleighPBX14 2645 /images/SeattlePBX12backup 10914 /images/LenovoX240Win7Pro3-28-18 4224 /images/AtlantaPBX14 1511 /images/FreePBX14Generic 400764 /imagesmd5sum /images/DaytonPBX14/*
Park City Master Node:
1e94d361fbba4d85b29d96f712e32fd9 /images/DaytonPBX14/d1.fixed_size_partitions d41d8cd98f00b204e9800998ecf8427e /images/DaytonPBX14/d1.has_grub 4993d426db2dee227e855e92d71ccaa8 /images/DaytonPBX14/d1.mbr 12858c05d8f5d17d91923f128196688e /images/DaytonPBX14/d1.minimum.partitions 917677f5661fd3fe50c286f345fccbad /images/DaytonPBX14/d1.original.fstypes 80bd7d28edabbd6f8bf1b3a1f57c4642 /images/DaytonPBX14/d1.original.swapuuids 85ba8c44919a82093f77daa4f41daba6 /images/DaytonPBX14/d1p1.img 9a666c1cf93b22530ea018bfddf88b68 /images/DaytonPBX14/d1p2.img b57dd15de0e672a846db773a62e59f5d /images/DaytonPBX14/d1p3.img af82072fe7eda65d7cacd34dd792f6fb /images/DaytonPBX14/d1p4.ebr bf619eac0cdf3f68d496ea9344137e8b /images/DaytonPBX14/d1p5.ebr fd0c4ed9f768eebfbe4a8461063083de /images/DaytonPBX14/d1.partitionsPark City Storage Node:
1e94d361fbba4d85b29d96f712e32fd9 /images/DaytonPBX14/d1.fixed_size_partitions d41d8cd98f00b204e9800998ecf8427e /images/DaytonPBX14/d1.has_grub 9c6c7cfabd9ff42fb6913268284f626b /images/DaytonPBX14/d1.mbr 1ed8ce4056cf877bb780d8dfd67ebc4c /images/DaytonPBX14/d1.minimum.partitions 917677f5661fd3fe50c286f345fccbad /images/DaytonPBX14/d1.original.fstypes 80bd7d28edabbd6f8bf1b3a1f57c4642 /images/DaytonPBX14/d1.original.swapuuids a52b9eb48ce72559c3bc402b8b4926dc /images/DaytonPBX14/d1p1.img 27e0d1cd1f2b3cc07eee026bb5c20a77 /images/DaytonPBX14/d1p2.img cfd9d3831190fd003f586bd2da483c7e /images/DaytonPBX14/d1p3.img af82072fe7eda65d7cacd34dd792f6fb /images/DaytonPBX14/d1p4.ebr bf619eac0cdf3f68d496ea9344137e8b /images/DaytonPBX14/d1p5.ebr fd0c4ed9f768eebfbe4a8461063083de /images/DaytonPBX14/d1.partitionsBuffalo Storage Node:
1e94d361fbba4d85b29d96f712e32fd9 /images/DaytonPBX14/d1.fixed_size_partitions d41d8cd98f00b204e9800998ecf8427e /images/DaytonPBX14/d1.has_grub 4993d426db2dee227e855e92d71ccaa8 /images/DaytonPBX14/d1.mbr 12858c05d8f5d17d91923f128196688e /images/DaytonPBX14/d1.minimum.partitions 917677f5661fd3fe50c286f345fccbad /images/DaytonPBX14/d1.original.fstypes 80bd7d28edabbd6f8bf1b3a1f57c4642 /images/DaytonPBX14/d1.original.swapuuids 85ba8c44919a82093f77daa4f41daba6 /images/DaytonPBX14/d1p1.img 9a666c1cf93b22530ea018bfddf88b68 /images/DaytonPBX14/d1p2.img b57dd15de0e672a846db773a62e59f5d /images/DaytonPBX14/d1p3.img af82072fe7eda65d7cacd34dd792f6fb /images/DaytonPBX14/d1p4.ebr bf619eac0cdf3f68d496ea9344137e8b /images/DaytonPBX14/d1p5.ebr fd0c4ed9f768eebfbe4a8461063083de /images/DaytonPBX14/d1.partitionsDallas Storage Node:
1e94d361fbba4d85b29d96f712e32fd9 /images/DaytonPBX14/d1.fixed_size_partitions d41d8cd98f00b204e9800998ecf8427e /images/DaytonPBX14/d1.has_grub 9c6c7cfabd9ff42fb6913268284f626b /images/DaytonPBX14/d1.mbr 1ed8ce4056cf877bb780d8dfd67ebc4c /images/DaytonPBX14/d1.minimum.partitions 917677f5661fd3fe50c286f345fccbad /images/DaytonPBX14/d1.original.fstypes 80bd7d28edabbd6f8bf1b3a1f57c4642 /images/DaytonPBX14/d1.original.swapuuids b0d9859b58422eb99ea09444ee6129dd /images/DaytonPBX14/d1p1.img 9a666c1cf93b22530ea018bfddf88b68 /images/DaytonPBX14/d1p2.img b57dd15de0e672a846db773a62e59f5d /images/DaytonPBX14/d1p3.img af82072fe7eda65d7cacd34dd792f6fb /images/DaytonPBX14/d1p4.ebr bf619eac0cdf3f68d496ea9344137e8b /images/DaytonPBX14/d1p5.ebr fd0c4ed9f768eebfbe4a8461063083de /images/DaytonPBX14/d1.partitions -
@kafluke Looking through the information you posted it seems like the Buffalo storage is pretty much in sync. Only difference at least in the size output is image
WindowsServer2016Genericandbigsnapins- the later might not be an image anyways. But the other two nodes are definitely not in sync.While trying to figure out some other replication hickup I found and fixed a couple of issues within the code. All that will be in the next release. Hopefully coming soon. Would be interesting to test those changes in an environment like yours. Are you keen to give it a try?
-
You bet! Just let me know when the update is out! Yeah, it looks like buffalo is the only one syncing fully. I had to manually sync the other two nodes.
-
@mronh The current changes are on a new branch
replication(link) which I will merge intoworkingafter a first round of feedback. It will all be in the next official release.Not sure if you have ever installed FOG unstable/testing. This is done using git to checkout the current code and install from that.
git clone https://github.com/FOGProject/fogproject/ cd fogproject git checkout replication cd bin ./installfog.shImportant notice: I had to change some of the hashing code too and therefore nodes being on different versions (1.5.4 or working VS. replication branch) will end up replicating images over and over again. So you need to have all nodes on the replication branch or setup up a separate test environment!!
Please make sure you stop replication first (
systemctl stop FOGImageReplicator), then update the storage node and after that update master node. -
@Sebastian-Roth This is what I get when I try and checkout replication:
error: pathspec ‘replication’ did not match any file(s) known to git.
-
@Sebastian-Roth never mind. I had to remove what I had checked out already “rm -rf /fogproject”
-
@Sebastian-Roth Okay I updated all 3 nodes to the “replication” branch and then I updated the master and restarted the sync service.
-
@Sebastian-Roth It seems to be properly sync-ing now. I’ll check it tomorrow after a day of sinc-ing and let you know the results. The Dallas FOG Node is definitely growing in data
-
@kafluke Thanks for the feedback so far. Keep an eye on the logs and let us know if you feel this is working properly.
You might want to do a quick file size check on all your nodes tomorrow:
du -k --max-depth=1 /images | sort -k2 > /tmp/imagelist_xxx_node.txtDo that on all four nodes and copy those files over to your PC. Then use some kind of diff tool (e.g. www.quickdiff.com) to compare those file contents.
-
@Sebastian-Roth It’s definitely working now. I’m watching the hard drive space on my Dallas node steadily go up. Thanks for fixing this!
-
@Sebastian-Roth Ok it worked. All nodes are now in sync with the master node! Thanks again for the help! Looking forward to the next release
-
@kafluke Thanks for the feedback! Do you see “file mismatches” anymore in the logs now?
Will mark this solved now. Those fixes will be merged into the working branch and from there go into the next release.
-
@Sebastian-Roth No more “file mismatches”. Thanks again.
-
@Sebastian-Roth Did you ever merge this into working branch? I had to start over from scratch on a new FOG server and wanted to know if I need to check out “replication” or “working”
-
@kafluke You’d want to checkout
dev-branch.Working is more or less a stand in for Alpha RC’s now.
Dev-branch is a bug fix branch
