FOG nodes not in sync

[kafluke]

Here is my setup.

FOG server version (and all Nodes) = 1.5.4
CentOS 7

I have 1 main server and then 3 storage nodes.
Park City Server is the master node and then there are three more sub nodes

• Park City Node
• Buffalo Node
• Dallas Node

The Buffalo Node seems to be in sync but the other two clearly aren’t. Here are screen shots from the dashboard. Shouldn’t all Used space be equal? I noticed that an image that I updated on the master node didn’t get replicated to Dallas when I imaged a computer there.

Here is my replication log:
https://pastebin.com/EjHuS5A8

[Sebastian Roth]

@kafluke Yeah we have seen similar replication issues reported with FOG 1.5.4 lately. I am still at it to figure out what exactly goes wrong. Will keep you up to date here. Possibly I will come back and ask you to test things.

[kafluke]

Sounds great

[Sebastian Roth]

@kafluke Re-visiting the log files and your screenshots I am wondering if we can get a size listing of the images from all your servers. Can you please run du -m --max-depth=1 /images on all four servers and post output here?

As well could you please run md5sum /images/DaytonPBX14/* on all four servers and post output here as well. For all the outputs please make sure you name it appropriately so we know exactly which is which. Thanks in advance.

[kafluke]

@Sebastian-Roth I’ll gather that info and post here. Just FYI I did a manual sync from Park City main server over to the Buffalo Node. I was going to do the same for the other two until it was working again.

[kafluke]

@Sebastian-Roth

du -m --max-depth=1 /images

Park City Master Node:

3198    /images/TulsaPBX
11681   /images/DellOptiplex7050Win10Pro3-1-18
18792   /images/DellPrec5530Win10ProUEFI9-27-18
10986   /images/DellOptiplex7040Win10Pro3-5-18
2386    /images/DenverPBX14
52357   /images/DellLatitude7280Win10Pro3-1-18
16665   /images/DellLatitudeE55709-12-18
8675    /images/DellLatitudeE6420Win7Pro3-20-18
24192   /images/LAPBX12Backup
4868    /images/OrlandoFreePBX12Backup
21612   /images/DellLatitudeE7250Win10Pro3-13-18
17532   /images/DellLatitude7290Win10Pro8-3-18
2770    /images/ClevelandFreePBX13Backup
956     /images/FreePBX12FreshInstall
6514    /images/RaleighPBX12
1730    /images/FresnoPBX14
11903   /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI
12237   /images/LenovoX240Win10Pro4-9-18
1679    /images/FreePBX13Generic
5338    /images/SuperMicroServer2012R2forNAS
3192    /images/Windows7ProGeneric64-bit
954     /images/FreshFreePBX12
1226    /images/FreshFreePBX13
2342    /images/SeattlePBX14
13003   /images/DellLatitude7290GenericTest
8444    /images/SuperMicroWin10ProVol1709forNAS
18261   /images/DellLatitude7490Win10Pro8-1-18
1259    /images/St.LouisPBX-12
1511    /images/FreePBX14Generic
3201    /images/bigsnapins
6650    /images/BostonPBX12backup
12387   /images/DellLatitude5290Win10Pro3-16-18
12979   /images/LenovoX230Updated
2594    /images/FreePBX14-1805-3
13403   /images/DellLatitudeE7270Win10Pro3-1-18
24049   /images/DellLatitude7280Win10Pro3-20-18
2520    /images/OrlandoFreePBX14
14793   /images/DellLatitude5480Win10Pro3-1-18
2501    /images/LAPBX14
2426    /images/RaleighPBX14
2645    /images/SeattlePBX12backup
3562    /images/DetroitPBXBackup13
4224    /images/AtlantaPBX14
1       /images/postdownloadscripts
2512    /images/BostonPBX14
19018   /images/DellAlienwareBaseWin10Pro9-21-18
26821   /images/DellLatitude5480Win10Pro6-4-18
31185   /images/Win-Pak
3980    /images/Win10ProGeneric
2584    /images/St.LouisPBX-14
3256    /images/13to14PBXupgradetest
5874    /images/Windows10Pro64bit1709Vol
10914   /images/LenovoX240Win7Pro3-28-18
2673    /images/dev
1948    /images/FreePBX13-Ext4-Base
1       /images/lost+found
18475   /images/DellLatitudeE5470Win10Pro3-8-18
7164    /images/Windows10Pro64bit1803Vol
25542   /images/DellLatitudeE5450Win10Pro3-1-18
4452    /images/WindowsServer2016Generic
2824    /images/PortlandPBX14
3436    /images/PortlandPBX12Backup
28037   /images/DellAlienwareDevelopersWin10Pro9-27-18
16440   /images/DellOptiplex9020Win10Pro5-25-18
2495    /images/DaytonPBX14
4695    /images/DaytonPBX12
618490  /images

Park City Storage Node:

3256    /images/13to14PBXupgradetest
11587   /images/DellLatitude7280Win10Pro3-20-18
8675    /images/DellLatitudeE6420Win7Pro3-20-18
1       /images/postdownloadscripts
9491    /images/DellOptiplex7050Win10Pro3-1-18
4452    /images/WindowsServer2016Generic
3980    /images/Win10ProGeneric
8444    /images/SuperMicroWin10ProVol1709forNAS
2485    /images/DaytonPBX14
1511    /images/FreePBX14Generic
1948    /images/FreePBX13-Ext4-Base
12237   /images/LenovoX240Win10Pro4-9-18
20576   /images/DellOptiplex9020Win10Pro5-25-18
4695    /images/DaytonPBX12
2645    /images/SeattlePBX12backup
1831    /images/FreePBX14-1805-3
5874    /images/Windows10Pro64bit1709Vol
12420   /images/DellLatitudeE5450Win10Pro3-1-18
956     /images/FreePBX12FreshInstall
2520    /images/OrlandoFreePBX14
31185   /images/Win-Pak
11903   /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI
11648   /images/DellLatitudeE7270Win10Pro3-1-18
5039    /images/AtlantaPBX14
2349    /images/RaleighPBX14
6650    /images/BostonPBX12backup
5338    /images/SuperMicroServer2012R2forNAS
3192    /images/Windows7ProGeneric64-bit
2770    /images/ClevelandFreePBX13Backup
3436    /images/PortlandPBX12Backup
2512    /images/BostonPBX14
3578    /images/DetroitPBXBackup13
11500   /images/DellOptiplex7040Win10Pro3-5-18
14565   /images/DellLatitudeE5470Win10Pro3-8-18
3509    /images/St.LouisPBX-14
2342    /images/SeattlePBX14
2501    /images/LAPBX14
1259    /images/St.LouisPBX-12
1       /images/dev
11372   /images/DellLatitudeE7250Win10Pro3-13-18
10914   /images/LenovoX240Win7Pro3-28-18
4868    /images/OrlandoFreePBX12Backup
6514    /images/RaleighPBX12
2573    /images/DenverPBX14
14793   /images/DellLatitude5480Win10Pro3-1-18
2824    /images/PortlandPBX14
298697  /images

Buffalo Storage Node:

12237   /images/LenovoX240Win10Pro4-9-18
956     /images/FreePBX12FreshInstall
4695    /images/DaytonPBX12
52357   /images/DellLatitude7280Win10Pro3-1-18
11681   /images/DellOptiplex7050Win10Pro3-1-18
2594    /images/FreePBX14-1805-3
2520    /images/OrlandoFreePBX14
2645    /images/SeattlePBX12backup
1226    /images/FreshFreePBX13
3562    /images/DetroitPBXBackup13
24192   /images/LAPBX12Backup
2584    /images/St.LouisPBX-14
1259    /images/St.LouisPBX-12
1730    /images/FresnoPBX14
10914   /images/LenovoX240Win7Pro3-28-18
5874    /images/Windows10Pro64bit1709Vol
10986   /images/DellOptiplex7040Win10Pro3-5-18
5338    /images/SuperMicroServer2012R2forNAS
2342    /images/SeattlePBX14
18261   /images/DellLatitude7490Win10Pro8-1-18
2386    /images/DenverPBX14
4868    /images/OrlandoFreePBX12Backup
16665   /images/DellLatitudeE55709-12-18
3198    /images/TulsaPBX
12979   /images/LenovoX230Updated
13403   /images/DellLatitudeE7270Win10Pro3-1-18
1679    /images/FreePBX13Generic
1       /images/bigsnapins
3436    /images/PortlandPBX12Backup
3192    /images/Windows7ProGeneric64-bit
14793   /images/DellLatitude5480Win10Pro3-1-18
6514    /images/RaleighPBX12
16440   /images/DellOptiplex9020Win10Pro5-25-18
2495    /images/DaytonPBX14
7607    /images/WindowsServer2016Generic
25542   /images/DellLatitudeE5450Win10Pro3-1-18
1       /images/dev
18475   /images/DellLatitudeE5470Win10Pro3-8-18
8675    /images/DellLatitudeE6420Win7Pro3-20-18
19018   /images/DellAlienwareBaseWin10Pro9-21-18
1       /images/lost+found
1511    /images/FreePBX14Generic
17532   /images/DellLatitude7290Win10Pro8-3-18
12387   /images/DellLatitude5290Win10Pro3-16-18
7164    /images/Windows10Pro64bit1803Vol
4224    /images/AtlantaPBX14
3980    /images/Win10ProGeneric
2512    /images/BostonPBX14
8444    /images/SuperMicroWin10ProVol1709forNAS
1948    /images/FreePBX13-Ext4-Base
2501    /images/LAPBX14
11903   /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI
6650    /images/BostonPBX12backup
21612   /images/DellLatitudeE7250Win10Pro3-13-18
2770    /images/ClevelandFreePBX13Backup
2426    /images/RaleighPBX14
954     /images/FreshFreePBX12
26821   /images/DellLatitude5480Win10Pro6-4-18
31185   /images/Win-Pak
2824    /images/PortlandPBX14
24049   /images/DellLatitude7280Win10Pro3-20-18
18792   /images/DellPrec5530Win10ProUEFI9-27-18
3256    /images/13to14PBXupgradetest
13003   /images/DellLatitude7290GenericTest
28037   /images/DellAlienwareDevelopersWin10Pro9-27-18
1       /images/postdownloadscripts
615772  /images

Dallas Storage Node:

12387   /images/DellLatitude5290Win10Pro3-16-18
1       /images/postdownloadscripts
14565   /images/DellLatitudeE5470Win10Pro3-8-18
12237   /images/LenovoX240Win10Pro4-9-18
19018   /images/DellAlienwareBaseWin10Pro9-21-18
1948    /images/FreePBX13-Ext4-Base
11372   /images/DellLatitudeE7250Win10Pro3-13-18
6514    /images/RaleighPBX12
5338    /images/SuperMicroServer2012R2forNAS
956     /images/FreePBX12FreshInstall
2594    /images/FreePBX14-1805-3
2501    /images/LAPBX14
3980    /images/Win10ProGeneric
20576   /images/DellOptiplex9020Win10Pro5-25-18
8444    /images/SuperMicroWin10ProVol1709forNAS
2386    /images/DenverPBX14
4868    /images/OrlandoFreePBX12Backup
24495   /images/DellLatitudeE5450Win10Pro3-1-18
11903   /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI
11587   /images/DellLatitude7280Win10Pro3-20-18
6650    /images/BostonPBX12backup
31185   /images/Win-Pak
21268   /images/DellLatitude7280Win10Pro3-1-18
5874    /images/Windows10Pro64bit1709Vol
28037   /images/DellAlienwareDevelopersWin10Pro9-27-18
9491    /images/DellOptiplex7050Win10Pro3-1-18
2584    /images/St.LouisPBX-14
2512    /images/BostonPBX14
11500   /images/DellOptiplex7040Win10Pro3-5-18
8675    /images/DellLatitudeE6420Win7Pro3-20-18
2520    /images/OrlandoFreePBX14
2342    /images/SeattlePBX14
4452    /images/WindowsServer2016Generic
14793   /images/DellLatitude5480Win10Pro3-1-18
3198    /images/TulsaPBX
3436    /images/PortlandPBX12Backup
11648   /images/DellLatitudeE7270Win10Pro3-1-18
2495    /images/DaytonPBX14
3192    /images/Windows7ProGeneric64-bit
1       /images/dev
2824    /images/PortlandPBX14
7164    /images/Windows10Pro64bit1803Vol
3256    /images/13to14PBXupgradetest
2770    /images/ClevelandFreePBX13Backup
4695    /images/DaytonPBX12
1259    /images/St.LouisPBX-12
3578    /images/DetroitPBXBackup13
2426    /images/RaleighPBX14
2645    /images/SeattlePBX12backup
10914   /images/LenovoX240Win7Pro3-28-18
4224    /images/AtlantaPBX14
1511    /images/FreePBX14Generic
400764  /images

md5sum /images/DaytonPBX14/*

Park City Master Node:

1e94d361fbba4d85b29d96f712e32fd9  /images/DaytonPBX14/d1.fixed_size_partitions
d41d8cd98f00b204e9800998ecf8427e  /images/DaytonPBX14/d1.has_grub
4993d426db2dee227e855e92d71ccaa8  /images/DaytonPBX14/d1.mbr
12858c05d8f5d17d91923f128196688e  /images/DaytonPBX14/d1.minimum.partitions
917677f5661fd3fe50c286f345fccbad  /images/DaytonPBX14/d1.original.fstypes
80bd7d28edabbd6f8bf1b3a1f57c4642  /images/DaytonPBX14/d1.original.swapuuids
85ba8c44919a82093f77daa4f41daba6  /images/DaytonPBX14/d1p1.img
9a666c1cf93b22530ea018bfddf88b68  /images/DaytonPBX14/d1p2.img
b57dd15de0e672a846db773a62e59f5d  /images/DaytonPBX14/d1p3.img
af82072fe7eda65d7cacd34dd792f6fb  /images/DaytonPBX14/d1p4.ebr
bf619eac0cdf3f68d496ea9344137e8b  /images/DaytonPBX14/d1p5.ebr
fd0c4ed9f768eebfbe4a8461063083de  /images/DaytonPBX14/d1.partitions

Park City Storage Node:

1e94d361fbba4d85b29d96f712e32fd9  /images/DaytonPBX14/d1.fixed_size_partitions
d41d8cd98f00b204e9800998ecf8427e  /images/DaytonPBX14/d1.has_grub
9c6c7cfabd9ff42fb6913268284f626b  /images/DaytonPBX14/d1.mbr
1ed8ce4056cf877bb780d8dfd67ebc4c  /images/DaytonPBX14/d1.minimum.partitions
917677f5661fd3fe50c286f345fccbad  /images/DaytonPBX14/d1.original.fstypes
80bd7d28edabbd6f8bf1b3a1f57c4642  /images/DaytonPBX14/d1.original.swapuuids
a52b9eb48ce72559c3bc402b8b4926dc  /images/DaytonPBX14/d1p1.img
27e0d1cd1f2b3cc07eee026bb5c20a77  /images/DaytonPBX14/d1p2.img
cfd9d3831190fd003f586bd2da483c7e  /images/DaytonPBX14/d1p3.img
af82072fe7eda65d7cacd34dd792f6fb  /images/DaytonPBX14/d1p4.ebr
bf619eac0cdf3f68d496ea9344137e8b  /images/DaytonPBX14/d1p5.ebr
fd0c4ed9f768eebfbe4a8461063083de  /images/DaytonPBX14/d1.partitions

Buffalo Storage Node:

1e94d361fbba4d85b29d96f712e32fd9  /images/DaytonPBX14/d1.fixed_size_partitions
d41d8cd98f00b204e9800998ecf8427e  /images/DaytonPBX14/d1.has_grub
4993d426db2dee227e855e92d71ccaa8  /images/DaytonPBX14/d1.mbr
12858c05d8f5d17d91923f128196688e  /images/DaytonPBX14/d1.minimum.partitions
917677f5661fd3fe50c286f345fccbad  /images/DaytonPBX14/d1.original.fstypes
80bd7d28edabbd6f8bf1b3a1f57c4642  /images/DaytonPBX14/d1.original.swapuuids
85ba8c44919a82093f77daa4f41daba6  /images/DaytonPBX14/d1p1.img
9a666c1cf93b22530ea018bfddf88b68  /images/DaytonPBX14/d1p2.img
b57dd15de0e672a846db773a62e59f5d  /images/DaytonPBX14/d1p3.img
af82072fe7eda65d7cacd34dd792f6fb  /images/DaytonPBX14/d1p4.ebr
bf619eac0cdf3f68d496ea9344137e8b  /images/DaytonPBX14/d1p5.ebr
fd0c4ed9f768eebfbe4a8461063083de  /images/DaytonPBX14/d1.partitions

Dallas Storage Node:

1e94d361fbba4d85b29d96f712e32fd9  /images/DaytonPBX14/d1.fixed_size_partitions
d41d8cd98f00b204e9800998ecf8427e  /images/DaytonPBX14/d1.has_grub
9c6c7cfabd9ff42fb6913268284f626b  /images/DaytonPBX14/d1.mbr
1ed8ce4056cf877bb780d8dfd67ebc4c  /images/DaytonPBX14/d1.minimum.partitions
917677f5661fd3fe50c286f345fccbad  /images/DaytonPBX14/d1.original.fstypes
80bd7d28edabbd6f8bf1b3a1f57c4642  /images/DaytonPBX14/d1.original.swapuuids
b0d9859b58422eb99ea09444ee6129dd  /images/DaytonPBX14/d1p1.img
9a666c1cf93b22530ea018bfddf88b68  /images/DaytonPBX14/d1p2.img
b57dd15de0e672a846db773a62e59f5d  /images/DaytonPBX14/d1p3.img
af82072fe7eda65d7cacd34dd792f6fb  /images/DaytonPBX14/d1p4.ebr
bf619eac0cdf3f68d496ea9344137e8b  /images/DaytonPBX14/d1p5.ebr
fd0c4ed9f768eebfbe4a8461063083de  /images/DaytonPBX14/d1.partitions

[Sebastian Roth]

@kafluke Looking through the information you posted it seems like the Buffalo storage is pretty much in sync. Only difference at least in the size output is image WindowsServer2016Generic and bigsnapins - the later might not be an image anyways. But the other two nodes are definitely not in sync.

While trying to figure out some other replication hickup I found and fixed a couple of issues within the code. All that will be in the next release. Hopefully coming soon. Would be interesting to test those changes in an environment like yours. Are you keen to give it a try?

[kafluke]

You bet! Just let me know when the update is out! Yeah, it looks like buffalo is the only one syncing fully. I had to manually sync the other two nodes.

[Sebastian Roth]

@mronh The current changes are on a new branch replication (link) which I will merge into working after a first round of feedback. It will all be in the next official release.

Not sure if you have ever installed FOG unstable/testing. This is done using git to checkout the current code and install from that.

git clone https://github.com/FOGProject/fogproject/
cd fogproject
git checkout replication
cd bin
./installfog.sh

Important notice: I had to change some of the hashing code too and therefore nodes being on different versions (1.5.4 or working VS. replication branch) will end up replicating images over and over again. So you need to have all nodes on the replication branch or setup up a separate test environment!!

Please make sure you stop replication first (systemctl stop FOGImageReplicator), then update the storage node and after that update master node.

[kafluke]

@Sebastian-Roth This is what I get when I try and checkout replication:

error: pathspec ‘replication’ did not match any file(s) known to git.

[kafluke]

@Sebastian-Roth never mind. I had to remove what I had checked out already “rm -rf /fogproject”

[kafluke]

@Sebastian-Roth Okay I updated all 3 nodes to the “replication” branch and then I updated the master and restarted the sync service.

[kafluke]

@Sebastian-Roth It seems to be properly sync-ing now. I’ll check it tomorrow after a day of sinc-ing and let you know the results. The Dallas FOG Node is definitely growing in data

[Sebastian Roth]

@kafluke Thanks for the feedback so far. Keep an eye on the logs and let us know if you feel this is working properly.

You might want to do a quick file size check on all your nodes tomorrow:

du -k --max-depth=1 /images | sort -k2 > /tmp/imagelist_xxx_node.txt

Do that on all four nodes and copy those files over to your PC. Then use some kind of diff tool (e.g. www.quickdiff.com) to compare those file contents.

[kafluke]

@Sebastian-Roth It’s definitely working now. I’m watching the hard drive space on my Dallas node steadily go up. Thanks for fixing this!

[kafluke]

@Sebastian-Roth Ok it worked. All nodes are now in sync with the master node! Thanks again for the help! Looking forward to the next release

[Sebastian Roth]

@kafluke Thanks for the feedback! Do you see “file mismatches” anymore in the logs now?

Will mark this solved now. Those fixes will be merged into the working branch and from there go into the next release.

[kafluke]

@Sebastian-Roth No more “file mismatches”. Thanks again.

[kafluke]

@Sebastian-Roth Did you ever merge this into working branch? I had to start over from scratch on a new FOG server and wanted to know if I need to check out “replication” or “working”

[Tom Elliott]

@kafluke You’d want to checkout dev-branch.

Working is more or less a stand in for Alpha RC’s now.

Dev-branch is a bug fix branch