FOG nodes not in sync



  • Here is my setup.

    FOG server version (and all Nodes) = 1.5.4
    CentOS 7

    I have 1 main server and then 3 storage nodes.
    Park City Server is the master node and then there are three more sub nodes

    • Park City Node
    • Buffalo Node
    • Dallas Node

    The Buffalo Node seems to be in sync but the other two clearly aren’t. Here are screen shots from the dashboard. Shouldn’t all Used space be equal? I noticed that an image that I updated on the master node didn’t get replicated to Dallas when I imaged a computer there.

    alt text
    alt text
    alt text
    alt text

    Here is my replication log:
    https://pastebin.com/EjHuS5A8


  • Senior Developer

    @kafluke You’d want to checkout dev-branch.

    Working is more or less a stand in for Alpha RC’s now.

    Dev-branch is a bug fix branch



  • @Sebastian-Roth Did you ever merge this into working branch? I had to start over from scratch on a new FOG server and wanted to know if I need to check out “replication” or “working”



  • @Sebastian-Roth No more “file mismatches”. Thanks again.


  • Developer

    @kafluke Thanks for the feedback! Do you see “file mismatches” anymore in the logs now?

    Will mark this solved now. Those fixes will be merged into the working branch and from there go into the next release.



  • @Sebastian-Roth Ok it worked. All nodes are now in sync with the master node! Thanks again for the help! Looking forward to the next release



  • @Sebastian-Roth It’s definitely working now. I’m watching the hard drive space on my Dallas node steadily go up. Thanks for fixing this!


  • Developer

    @kafluke Thanks for the feedback so far. Keep an eye on the logs and let us know if you feel this is working properly.

    You might want to do a quick file size check on all your nodes tomorrow:

    du -k --max-depth=1 /images | sort -k2 > /tmp/imagelist_xxx_node.txt
    

    Do that on all four nodes and copy those files over to your PC. Then use some kind of diff tool (e.g. www.quickdiff.com) to compare those file contents.



  • @Sebastian-Roth It seems to be properly sync-ing now. I’ll check it tomorrow after a day of sinc-ing and let you know the results. The Dallas FOG Node is definitely growing in data



  • @Sebastian-Roth Okay I updated all 3 nodes to the “replication” branch and then I updated the master and restarted the sync service.



  • @Sebastian-Roth never mind. I had to remove what I had checked out already “rm -rf /fogproject”



  • @Sebastian-Roth This is what I get when I try and checkout replication:

    error: pathspec ‘replication’ did not match any file(s) known to git.


  • Developer

    @mronh The current changes are on a new branch replication (link) which I will merge into working after a first round of feedback. It will all be in the next official release.

    Not sure if you have ever installed FOG unstable/testing. This is done using git to checkout the current code and install from that.

    git clone https://github.com/FOGProject/fogproject/
    cd fogproject
    git checkout replication
    cd bin
    ./installfog.sh
    

    Important notice: I had to change some of the hashing code too and therefore nodes being on different versions (1.5.4 or working VS. replication branch) will end up replicating images over and over again. So you need to have all nodes on the replication branch or setup up a separate test environment!!

    Please make sure you stop replication first (systemctl stop FOGImageReplicator), then update the storage node and after that update master node.



  • You bet! Just let me know when the update is out! Yeah, it looks like buffalo is the only one syncing fully. I had to manually sync the other two nodes.


  • Developer

    @kafluke Looking through the information you posted it seems like the Buffalo storage is pretty much in sync. Only difference at least in the size output is image WindowsServer2016Generic and bigsnapins - the later might not be an image anyways. But the other two nodes are definitely not in sync.

    While trying to figure out some other replication hickup I found and fixed a couple of issues within the code. All that will be in the next release. Hopefully coming soon. Would be interesting to test those changes in an environment like yours. Are you keen to give it a try?



  • @Sebastian-Roth

    du -m --max-depth=1 /images

    Park City Master Node:

    3198    /images/TulsaPBX
    11681   /images/DellOptiplex7050Win10Pro3-1-18
    18792   /images/DellPrec5530Win10ProUEFI9-27-18
    10986   /images/DellOptiplex7040Win10Pro3-5-18
    2386    /images/DenverPBX14
    52357   /images/DellLatitude7280Win10Pro3-1-18
    16665   /images/DellLatitudeE55709-12-18
    8675    /images/DellLatitudeE6420Win7Pro3-20-18
    24192   /images/LAPBX12Backup
    4868    /images/OrlandoFreePBX12Backup
    21612   /images/DellLatitudeE7250Win10Pro3-13-18
    17532   /images/DellLatitude7290Win10Pro8-3-18
    2770    /images/ClevelandFreePBX13Backup
    956     /images/FreePBX12FreshInstall
    6514    /images/RaleighPBX12
    1730    /images/FresnoPBX14
    11903   /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI
    12237   /images/LenovoX240Win10Pro4-9-18
    1679    /images/FreePBX13Generic
    5338    /images/SuperMicroServer2012R2forNAS
    3192    /images/Windows7ProGeneric64-bit
    954     /images/FreshFreePBX12
    1226    /images/FreshFreePBX13
    2342    /images/SeattlePBX14
    13003   /images/DellLatitude7290GenericTest
    8444    /images/SuperMicroWin10ProVol1709forNAS
    18261   /images/DellLatitude7490Win10Pro8-1-18
    1259    /images/St.LouisPBX-12
    1511    /images/FreePBX14Generic
    3201    /images/bigsnapins
    6650    /images/BostonPBX12backup
    12387   /images/DellLatitude5290Win10Pro3-16-18
    12979   /images/LenovoX230Updated
    2594    /images/FreePBX14-1805-3
    13403   /images/DellLatitudeE7270Win10Pro3-1-18
    24049   /images/DellLatitude7280Win10Pro3-20-18
    2520    /images/OrlandoFreePBX14
    14793   /images/DellLatitude5480Win10Pro3-1-18
    2501    /images/LAPBX14
    2426    /images/RaleighPBX14
    2645    /images/SeattlePBX12backup
    3562    /images/DetroitPBXBackup13
    4224    /images/AtlantaPBX14
    1       /images/postdownloadscripts
    2512    /images/BostonPBX14
    19018   /images/DellAlienwareBaseWin10Pro9-21-18
    26821   /images/DellLatitude5480Win10Pro6-4-18
    31185   /images/Win-Pak
    3980    /images/Win10ProGeneric
    2584    /images/St.LouisPBX-14
    3256    /images/13to14PBXupgradetest
    5874    /images/Windows10Pro64bit1709Vol
    10914   /images/LenovoX240Win7Pro3-28-18
    2673    /images/dev
    1948    /images/FreePBX13-Ext4-Base
    1       /images/lost+found
    18475   /images/DellLatitudeE5470Win10Pro3-8-18
    7164    /images/Windows10Pro64bit1803Vol
    25542   /images/DellLatitudeE5450Win10Pro3-1-18
    4452    /images/WindowsServer2016Generic
    2824    /images/PortlandPBX14
    3436    /images/PortlandPBX12Backup
    28037   /images/DellAlienwareDevelopersWin10Pro9-27-18
    16440   /images/DellOptiplex9020Win10Pro5-25-18
    2495    /images/DaytonPBX14
    4695    /images/DaytonPBX12
    618490  /images
    

    Park City Storage Node:

    3256    /images/13to14PBXupgradetest
    11587   /images/DellLatitude7280Win10Pro3-20-18
    8675    /images/DellLatitudeE6420Win7Pro3-20-18
    1       /images/postdownloadscripts
    9491    /images/DellOptiplex7050Win10Pro3-1-18
    4452    /images/WindowsServer2016Generic
    3980    /images/Win10ProGeneric
    8444    /images/SuperMicroWin10ProVol1709forNAS
    2485    /images/DaytonPBX14
    1511    /images/FreePBX14Generic
    1948    /images/FreePBX13-Ext4-Base
    12237   /images/LenovoX240Win10Pro4-9-18
    20576   /images/DellOptiplex9020Win10Pro5-25-18
    4695    /images/DaytonPBX12
    2645    /images/SeattlePBX12backup
    1831    /images/FreePBX14-1805-3
    5874    /images/Windows10Pro64bit1709Vol
    12420   /images/DellLatitudeE5450Win10Pro3-1-18
    956     /images/FreePBX12FreshInstall
    2520    /images/OrlandoFreePBX14
    31185   /images/Win-Pak
    11903   /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI
    11648   /images/DellLatitudeE7270Win10Pro3-1-18
    5039    /images/AtlantaPBX14
    2349    /images/RaleighPBX14
    6650    /images/BostonPBX12backup
    5338    /images/SuperMicroServer2012R2forNAS
    3192    /images/Windows7ProGeneric64-bit
    2770    /images/ClevelandFreePBX13Backup
    3436    /images/PortlandPBX12Backup
    2512    /images/BostonPBX14
    3578    /images/DetroitPBXBackup13
    11500   /images/DellOptiplex7040Win10Pro3-5-18
    14565   /images/DellLatitudeE5470Win10Pro3-8-18
    3509    /images/St.LouisPBX-14
    2342    /images/SeattlePBX14
    2501    /images/LAPBX14
    1259    /images/St.LouisPBX-12
    1       /images/dev
    11372   /images/DellLatitudeE7250Win10Pro3-13-18
    10914   /images/LenovoX240Win7Pro3-28-18
    4868    /images/OrlandoFreePBX12Backup
    6514    /images/RaleighPBX12
    2573    /images/DenverPBX14
    14793   /images/DellLatitude5480Win10Pro3-1-18
    2824    /images/PortlandPBX14
    298697  /images
    

    Buffalo Storage Node:

    12237   /images/LenovoX240Win10Pro4-9-18
    956     /images/FreePBX12FreshInstall
    4695    /images/DaytonPBX12
    52357   /images/DellLatitude7280Win10Pro3-1-18
    11681   /images/DellOptiplex7050Win10Pro3-1-18
    2594    /images/FreePBX14-1805-3
    2520    /images/OrlandoFreePBX14
    2645    /images/SeattlePBX12backup
    1226    /images/FreshFreePBX13
    3562    /images/DetroitPBXBackup13
    24192   /images/LAPBX12Backup
    2584    /images/St.LouisPBX-14
    1259    /images/St.LouisPBX-12
    1730    /images/FresnoPBX14
    10914   /images/LenovoX240Win7Pro3-28-18
    5874    /images/Windows10Pro64bit1709Vol
    10986   /images/DellOptiplex7040Win10Pro3-5-18
    5338    /images/SuperMicroServer2012R2forNAS
    2342    /images/SeattlePBX14
    18261   /images/DellLatitude7490Win10Pro8-1-18
    2386    /images/DenverPBX14
    4868    /images/OrlandoFreePBX12Backup
    16665   /images/DellLatitudeE55709-12-18
    3198    /images/TulsaPBX
    12979   /images/LenovoX230Updated
    13403   /images/DellLatitudeE7270Win10Pro3-1-18
    1679    /images/FreePBX13Generic
    1       /images/bigsnapins
    3436    /images/PortlandPBX12Backup
    3192    /images/Windows7ProGeneric64-bit
    14793   /images/DellLatitude5480Win10Pro3-1-18
    6514    /images/RaleighPBX12
    16440   /images/DellOptiplex9020Win10Pro5-25-18
    2495    /images/DaytonPBX14
    7607    /images/WindowsServer2016Generic
    25542   /images/DellLatitudeE5450Win10Pro3-1-18
    1       /images/dev
    18475   /images/DellLatitudeE5470Win10Pro3-8-18
    8675    /images/DellLatitudeE6420Win7Pro3-20-18
    19018   /images/DellAlienwareBaseWin10Pro9-21-18
    1       /images/lost+found
    1511    /images/FreePBX14Generic
    17532   /images/DellLatitude7290Win10Pro8-3-18
    12387   /images/DellLatitude5290Win10Pro3-16-18
    7164    /images/Windows10Pro64bit1803Vol
    4224    /images/AtlantaPBX14
    3980    /images/Win10ProGeneric
    2512    /images/BostonPBX14
    8444    /images/SuperMicroWin10ProVol1709forNAS
    1948    /images/FreePBX13-Ext4-Base
    2501    /images/LAPBX14
    11903   /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI
    6650    /images/BostonPBX12backup
    21612   /images/DellLatitudeE7250Win10Pro3-13-18
    2770    /images/ClevelandFreePBX13Backup
    2426    /images/RaleighPBX14
    954     /images/FreshFreePBX12
    26821   /images/DellLatitude5480Win10Pro6-4-18
    31185   /images/Win-Pak
    2824    /images/PortlandPBX14
    24049   /images/DellLatitude7280Win10Pro3-20-18
    18792   /images/DellPrec5530Win10ProUEFI9-27-18
    3256    /images/13to14PBXupgradetest
    13003   /images/DellLatitude7290GenericTest
    28037   /images/DellAlienwareDevelopersWin10Pro9-27-18
    1       /images/postdownloadscripts
    615772  /images
    

    Dallas Storage Node:

    12387   /images/DellLatitude5290Win10Pro3-16-18
    1       /images/postdownloadscripts
    14565   /images/DellLatitudeE5470Win10Pro3-8-18
    12237   /images/LenovoX240Win10Pro4-9-18
    19018   /images/DellAlienwareBaseWin10Pro9-21-18
    1948    /images/FreePBX13-Ext4-Base
    11372   /images/DellLatitudeE7250Win10Pro3-13-18
    6514    /images/RaleighPBX12
    5338    /images/SuperMicroServer2012R2forNAS
    956     /images/FreePBX12FreshInstall
    2594    /images/FreePBX14-1805-3
    2501    /images/LAPBX14
    3980    /images/Win10ProGeneric
    20576   /images/DellOptiplex9020Win10Pro5-25-18
    8444    /images/SuperMicroWin10ProVol1709forNAS
    2386    /images/DenverPBX14
    4868    /images/OrlandoFreePBX12Backup
    24495   /images/DellLatitudeE5450Win10Pro3-1-18
    11903   /images/DellLatitude52902-in1Win10Pro4-19-18-UEFI
    11587   /images/DellLatitude7280Win10Pro3-20-18
    6650    /images/BostonPBX12backup
    31185   /images/Win-Pak
    21268   /images/DellLatitude7280Win10Pro3-1-18
    5874    /images/Windows10Pro64bit1709Vol
    28037   /images/DellAlienwareDevelopersWin10Pro9-27-18
    9491    /images/DellOptiplex7050Win10Pro3-1-18
    2584    /images/St.LouisPBX-14
    2512    /images/BostonPBX14
    11500   /images/DellOptiplex7040Win10Pro3-5-18
    8675    /images/DellLatitudeE6420Win7Pro3-20-18
    2520    /images/OrlandoFreePBX14
    2342    /images/SeattlePBX14
    4452    /images/WindowsServer2016Generic
    14793   /images/DellLatitude5480Win10Pro3-1-18
    3198    /images/TulsaPBX
    3436    /images/PortlandPBX12Backup
    11648   /images/DellLatitudeE7270Win10Pro3-1-18
    2495    /images/DaytonPBX14
    3192    /images/Windows7ProGeneric64-bit
    1       /images/dev
    2824    /images/PortlandPBX14
    7164    /images/Windows10Pro64bit1803Vol
    3256    /images/13to14PBXupgradetest
    2770    /images/ClevelandFreePBX13Backup
    4695    /images/DaytonPBX12
    1259    /images/St.LouisPBX-12
    3578    /images/DetroitPBXBackup13
    2426    /images/RaleighPBX14
    2645    /images/SeattlePBX12backup
    10914   /images/LenovoX240Win7Pro3-28-18
    4224    /images/AtlantaPBX14
    1511    /images/FreePBX14Generic
    400764  /images
    

    md5sum /images/DaytonPBX14/*

    Park City Master Node:

    1e94d361fbba4d85b29d96f712e32fd9  /images/DaytonPBX14/d1.fixed_size_partitions
    d41d8cd98f00b204e9800998ecf8427e  /images/DaytonPBX14/d1.has_grub
    4993d426db2dee227e855e92d71ccaa8  /images/DaytonPBX14/d1.mbr
    12858c05d8f5d17d91923f128196688e  /images/DaytonPBX14/d1.minimum.partitions
    917677f5661fd3fe50c286f345fccbad  /images/DaytonPBX14/d1.original.fstypes
    80bd7d28edabbd6f8bf1b3a1f57c4642  /images/DaytonPBX14/d1.original.swapuuids
    85ba8c44919a82093f77daa4f41daba6  /images/DaytonPBX14/d1p1.img
    9a666c1cf93b22530ea018bfddf88b68  /images/DaytonPBX14/d1p2.img
    b57dd15de0e672a846db773a62e59f5d  /images/DaytonPBX14/d1p3.img
    af82072fe7eda65d7cacd34dd792f6fb  /images/DaytonPBX14/d1p4.ebr
    bf619eac0cdf3f68d496ea9344137e8b  /images/DaytonPBX14/d1p5.ebr
    fd0c4ed9f768eebfbe4a8461063083de  /images/DaytonPBX14/d1.partitions
    

    Park City Storage Node:

    1e94d361fbba4d85b29d96f712e32fd9  /images/DaytonPBX14/d1.fixed_size_partitions
    d41d8cd98f00b204e9800998ecf8427e  /images/DaytonPBX14/d1.has_grub
    9c6c7cfabd9ff42fb6913268284f626b  /images/DaytonPBX14/d1.mbr
    1ed8ce4056cf877bb780d8dfd67ebc4c  /images/DaytonPBX14/d1.minimum.partitions
    917677f5661fd3fe50c286f345fccbad  /images/DaytonPBX14/d1.original.fstypes
    80bd7d28edabbd6f8bf1b3a1f57c4642  /images/DaytonPBX14/d1.original.swapuuids
    a52b9eb48ce72559c3bc402b8b4926dc  /images/DaytonPBX14/d1p1.img
    27e0d1cd1f2b3cc07eee026bb5c20a77  /images/DaytonPBX14/d1p2.img
    cfd9d3831190fd003f586bd2da483c7e  /images/DaytonPBX14/d1p3.img
    af82072fe7eda65d7cacd34dd792f6fb  /images/DaytonPBX14/d1p4.ebr
    bf619eac0cdf3f68d496ea9344137e8b  /images/DaytonPBX14/d1p5.ebr
    fd0c4ed9f768eebfbe4a8461063083de  /images/DaytonPBX14/d1.partitions
    

    Buffalo Storage Node:

    1e94d361fbba4d85b29d96f712e32fd9  /images/DaytonPBX14/d1.fixed_size_partitions
    d41d8cd98f00b204e9800998ecf8427e  /images/DaytonPBX14/d1.has_grub
    4993d426db2dee227e855e92d71ccaa8  /images/DaytonPBX14/d1.mbr
    12858c05d8f5d17d91923f128196688e  /images/DaytonPBX14/d1.minimum.partitions
    917677f5661fd3fe50c286f345fccbad  /images/DaytonPBX14/d1.original.fstypes
    80bd7d28edabbd6f8bf1b3a1f57c4642  /images/DaytonPBX14/d1.original.swapuuids
    85ba8c44919a82093f77daa4f41daba6  /images/DaytonPBX14/d1p1.img
    9a666c1cf93b22530ea018bfddf88b68  /images/DaytonPBX14/d1p2.img
    b57dd15de0e672a846db773a62e59f5d  /images/DaytonPBX14/d1p3.img
    af82072fe7eda65d7cacd34dd792f6fb  /images/DaytonPBX14/d1p4.ebr
    bf619eac0cdf3f68d496ea9344137e8b  /images/DaytonPBX14/d1p5.ebr
    fd0c4ed9f768eebfbe4a8461063083de  /images/DaytonPBX14/d1.partitions
    

    Dallas Storage Node:

    1e94d361fbba4d85b29d96f712e32fd9  /images/DaytonPBX14/d1.fixed_size_partitions
    d41d8cd98f00b204e9800998ecf8427e  /images/DaytonPBX14/d1.has_grub
    9c6c7cfabd9ff42fb6913268284f626b  /images/DaytonPBX14/d1.mbr
    1ed8ce4056cf877bb780d8dfd67ebc4c  /images/DaytonPBX14/d1.minimum.partitions
    917677f5661fd3fe50c286f345fccbad  /images/DaytonPBX14/d1.original.fstypes
    80bd7d28edabbd6f8bf1b3a1f57c4642  /images/DaytonPBX14/d1.original.swapuuids
    b0d9859b58422eb99ea09444ee6129dd  /images/DaytonPBX14/d1p1.img
    9a666c1cf93b22530ea018bfddf88b68  /images/DaytonPBX14/d1p2.img
    b57dd15de0e672a846db773a62e59f5d  /images/DaytonPBX14/d1p3.img
    af82072fe7eda65d7cacd34dd792f6fb  /images/DaytonPBX14/d1p4.ebr
    bf619eac0cdf3f68d496ea9344137e8b  /images/DaytonPBX14/d1p5.ebr
    fd0c4ed9f768eebfbe4a8461063083de  /images/DaytonPBX14/d1.partitions
    


  • @Sebastian-Roth I’ll gather that info and post here. Just FYI I did a manual sync from Park City main server over to the Buffalo Node. I was going to do the same for the other two until it was working again.


  • Developer

    @kafluke Re-visiting the log files and your screenshots I am wondering if we can get a size listing of the images from all your servers. Can you please run du -m --max-depth=1 /images on all four servers and post output here?

    As well could you please run md5sum /images/DaytonPBX14/* on all four servers and post output here as well. For all the outputs please make sure you name it appropriately so we know exactly which is which. Thanks in advance.



  • Sounds great


  • Developer

    @kafluke Yeah we have seen similar replication issues reported with FOG 1.5.4 lately. I am still at it to figure out what exactly goes wrong. Will keep you up to date here. Possibly I will come back and ask you to test things.


Log in to reply
 

391
Online

5.8k
Users

13.1k
Topics

123.3k
Posts