Replication problems 1.5.4 - always copying

[jflippen]

@Sebastian-Roth Definitely! Though I will probably test on some VM’s first since our actual FOG server is running scripts to back everything up to our nodes instead since we were having issues with both replication services. Will the test code be on the -working channel or a different one? I’ll try to provide feedback when I can. There are other bugs I need to report on as well, but haven’t had time to document.

[Sebastian Roth]

@jflippen The current changes are on a new branch replication (link) which I will merge into working after a first round of feedback.

Important notice: I had to change some of the hashing code too and therefore nodes being on different versions (1.5.4 or working VS. replication branch) will end up replicating images over and over again. So you need to have all nodes on the replication branch or setup up a separate test environment!!

Please make sure you stop replication first (systemctl stop FOGImageReplicator), then update the storage node and after that update master node.

[jflippen]

@Sebastian-Roth Okay, so I managed to test out the new branch at home on my VM’s (like I mentioned before, not going to touch our primary FOG server right now sense we have a work around). Here is what I found:

• The first time around I tried just doing an in-place upgrade of the working 1.6 branch. This, however seemed to not work properly as it did not change anything after I swapped branches and installed on all three (stopping the service and rebooting on each VM, going from nodes to master).

• The 2nd time I did a fresh install of Centos 7 on each VM, followed the wiki guide, and went directly to the replication branch. I then created a fake image by creating a folder and then putting some random files in there (I don’t have PXE boot or a server environment at home). When I checked the log and did some testing, it looks like it’s working! Previously, FOG would not delete mismatched files, meaning the files would never get updated on the node, but would keep trying to run an lftp sync… It would even say files didn’t match even if they had the same md5sum at times. This seems to have been fixed in your new branch! I have included my log below.

• During testing I noticed that the replication service won’t remove files that are on the node but not the master (from changing the files I was testing with on the master). Would it be possible to implement an if exist variable that checks an array of the file names in an image folder against the master? This would be handy for keeping the driver injection folder we use in the images folder synced properly too when removing deprecated models.

The Log, as promised

[11-07-18 6:34:23 am] 
==================================
===        ====    =====      ====
===  =========  ==  ===   ==   ===
===  ========  ====  ==  ====  ===
===  ========  ====  ==  =========
===      ====  ====  ==  =========
===  ========  ====  ==  ===   ===
===  ========  ====  ==  ====  ===
===  =========  ==  ===   ==   ===
===  ==========    =====      ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[11-07-18 6:34:23 am] Interface Ready with IP Address: 127.0.0.1
[11-07-18 6:34:23 am] Interface Ready with IP Address: 127.0.1.1
[11-07-18 6:34:23 am] Interface Ready with IP Address: 192.168.1.16
[11-07-18 6:34:23 am] Interface Ready with IP Address: FOG-MASTER
[11-07-18 6:34:23 am] FOGService: ImageReplicator - Waiting for mysql to be available
[11-07-18 6:34:35 am]  * Starting ImageReplicator Service
[11-07-18 6:34:35 am]  * Checking for new items every 600 seconds
[11-07-18 6:34:35 am]  * Starting service loop
[11-07-18 6:34:36 am]  * Starting Image Replication.
[11-07-18 6:34:36 am]  * We are group ID: 1. We are group name: default
[11-07-18 6:34:36 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 6:34:36 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 6:34:36 am]  | Replicating postdownloadscripts
[11-07-18 6:34:38 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:34:38 am]  | File Name: postdownloadscripts
[11-07-18 6:34:38 am]  | NODE-1 server does not appear to be online.
[11-07-18 6:34:38 am]  | NODE-2 server does not appear to be online.
[11-07-18 6:34:38 am]  | Replicating postinitscripts
[11-07-18 6:34:40 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:34:40 am]  | File Name: dev/postinitscripts
[11-07-18 6:34:40 am]  | NODE-1 server does not appear to be online.
[11-07-18 6:34:40 am]  | NODE-2 server does not appear to be online.
[11-07-18 6:34:40 am]  * Not syncing Image between groups
[11-07-18 6:34:40 am]  | Image Name: test
[11-07-18 6:34:40 am]  | There are no other members to sync to.
[11-07-18 6:34:40 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 6:34:42 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:34:42 am]  | Image Name: test
[11-07-18 6:34:42 am]  | NODE-1 server does not appear to be online.
[11-07-18 6:34:42 am]  | NODE-2 server does not appear to be online.
[11-07-18 6:44:42 am]  * Starting Image Replication.
[11-07-18 6:44:42 am]  * We are group ID: 1. We are group name: default
[11-07-18 6:44:42 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 6:44:42 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 6:44:42 am]  | Replicating postdownloadscripts
[11-07-18 6:44:42 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:44:42 am]  | File Name: postdownloadscripts
[11-07-18 6:44:43 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-07-18 6:44:43 am]  * All files synced for this item.
[11-07-18 6:44:43 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-07-18 6:44:43 am]  * All files synced for this item.
[11-07-18 6:44:43 am]  | Replicating postinitscripts
[11-07-18 6:44:44 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:44:44 am]  | File Name: dev/postinitscripts
[11-07-18 6:44:44 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-07-18 6:44:44 am]  * All files synced for this item.
[11-07-18 6:44:45 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-07-18 6:44:45 am]  * All files synced for this item.
[11-07-18 6:44:45 am]  * Not syncing Image between groups
[11-07-18 6:44:45 am]  | Image Name: test
[11-07-18 6:44:45 am]  | There are no other members to sync to.
[11-07-18 6:44:45 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 6:44:46 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:44:46 am]  | Image Name: test
[11-07-18 6:44:46 am]   # test: File does not exist 2014-10-31 11.52.32.jpg (NODE-1)
[11-07-18 6:44:46 am]   # test: File size mismatch - CentOS-7-x86_64-DVD-1804.iso: 4470079488 != 2903506944
[11-07-18 6:44:46 am]   # test: Deleting remote file CentOS-7-x86_64-DVD-1804.iso
[11-07-18 6:44:46 am]   # test: File does not exist Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-07-18 6:44:46 am]   # test: File does not exist FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-07-18 6:44:46 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.test.transfer.NODE-1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.17
[11-07-18 6:44:46 am]  | Started sync for Image test - Resource id #714
[11-07-18 6:44:46 am]   # test: File does not exist 2014-10-31 11.52.32.jpg (NODE-2)
[11-07-18 6:44:46 am]   # test: File size mismatch - CentOS-7-x86_64-DVD-1804.iso: 4470079488 != 2903506944
[11-07-18 6:44:46 am]   # test: Deleting remote file CentOS-7-x86_64-DVD-1804.iso
[11-07-18 6:44:46 am]   # test: File does not exist Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-07-18 6:44:46 am]   # test: File does not exist FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-07-18 6:44:46 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator.test.transfer.NODE-2.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.18
[11-07-18 6:44:46 am]  | Started sync for Image test - Resource id #729
[11-07-18 6:48:10 am]  | Sync finished - Resource id #714
[11-07-18 6:48:11 am]  | Sync finished - Resource id #729
[11-07-18 6:54:42 am]  * Starting Image Replication.
[11-07-18 6:54:42 am]  * We are group ID: 1. We are group name: default
[11-07-18 6:54:42 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 6:54:42 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 6:54:42 am]  | Replicating postdownloadscripts
[11-07-18 6:54:43 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:54:43 am]  | File Name: postdownloadscripts
[11-07-18 6:54:43 am]   # postdownloadscripts: File does not exist fog.custominstall (NODE-1)
[11-07-18 6:54:43 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-07-18 6:54:43 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.17
[11-07-18 6:54:43 am]  | Started sync for Image postdownloadscripts - Resource id #781
[11-07-18 6:54:44 am]   # postdownloadscripts: File does not exist fog.custominstall (NODE-2)
[11-07-18 6:54:44 am]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-07-18 6:54:44 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-2.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.18
[11-07-18 6:54:44 am]  | Started sync for Image postdownloadscripts - Resource id #794
[11-07-18 6:54:44 am]  | Replicating postinitscripts
[11-07-18 6:54:45 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:54:45 am]  | File Name: dev/postinitscripts
[11-07-18 6:54:45 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-07-18 6:54:45 am]  * All files synced for this item.
[11-07-18 6:54:45 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-07-18 6:54:45 am]  * All files synced for this item.
[11-07-18 6:54:46 am]  * Not syncing Image between groups
[11-07-18 6:54:46 am]  | Image Name: test
[11-07-18 6:54:46 am]  | There are no other members to sync to.
[11-07-18 6:54:46 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 6:54:46 am]  * Found Image to transfer to 2 nodes
[11-07-18 6:54:46 am]  | Image Name: test
[11-07-18 6:54:47 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-07-18 6:54:47 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-1)
[11-07-18 6:54:48 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-07-18 6:54:48 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-07-18 6:54:48 am]  * All files synced for this item.
[11-07-18 6:54:49 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-07-18 6:54:49 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-2)
[11-07-18 6:54:49 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-07-18 6:54:50 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-07-18 6:54:50 am]  * All files synced for this item.
[11-07-18 6:54:50 am]  | Sync finished - Resource id #781
[11-07-18 6:54:50 am]  | Sync finished - Resource id #794
[11-07-18 7:04:42 am]  * Starting Image Replication.
[11-07-18 7:04:42 am]  * We are group ID: 1. We are group name: default
[11-07-18 7:04:42 am]  * We are node ID: 1. We are node name: DefaultMember
[11-07-18 7:04:42 am]  * Attempting to perform Group -> Group image replication.
[11-07-18 7:04:42 am]  | Replicating postdownloadscripts
[11-07-18 7:04:42 am]  * Found Image to transfer to 2 nodes
[11-07-18 7:04:42 am]  | File Name: postdownloadscripts
[11-07-18 7:04:42 am]   # postdownloadscripts: File does not exist fog.copydrivers (NODE-1)
[11-07-18 7:04:42 am]   # postdownloadscripts: File size mismatch - fog.custominstall: 1673 != 235
[11-07-18 7:04:42 am]   # postdownloadscripts: Deleting remote file fog.custominstall
[11-07-18 7:04:43 am]   # postdownloadscripts: File size mismatch - fog.postdownload: 271 != 235
[11-07-18 7:04:43 am]   # postdownloadscripts: Deleting remote file fog.postdownload
[11-07-18 7:04:43 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-1.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.17
[11-07-18 7:04:43 am]  | Started sync for Image postdownloadscripts - Resource id #975
[11-07-18 7:04:43 am]   # postdownloadscripts: File does not exist fog.copydrivers (NODE-2)
[11-07-18 7:04:43 am]   # postdownloadscripts: File size mismatch - fog.custominstall: 1673 != 235
[11-07-18 7:04:43 am]   # postdownloadscripts: Deleting remote file fog.custominstall
[11-07-18 7:04:43 am]   # postdownloadscripts: File size mismatch - fog.postdownload: 271 != 235
[11-07-18 7:04:43 am]   # postdownloadscripts: Deleting remote file fog.postdownload
[11-07-18 7:04:43 am]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogreplicator..transfer.NODE-2.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/postdownloadscripts" "/images/postdownloadscripts"; exit' -u fog,[Protected] 192.168.1.18
[11-07-18 7:04:43 am]  | Started sync for Image postdownloadscripts - Resource id #993
[11-07-18 7:04:43 am]  | Replicating postinitscripts
[11-07-18 7:04:44 am]  * Found Image to transfer to 2 nodes
[11-07-18 7:04:44 am]  | File Name: dev/postinitscripts
[11-07-18 7:04:44 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-07-18 7:04:44 am]  * All files synced for this item.
[11-07-18 7:04:44 am]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-07-18 7:04:44 am]  * All files synced for this item.
[11-07-18 7:04:45 am]  * Not syncing Image between groups
[11-07-18 7:04:45 am]  | Image Name: test
[11-07-18 7:04:45 am]  | There are no other members to sync to.
[11-07-18 7:04:45 am]  * Attempting to perform Group -> Nodes image replication.
[11-07-18 7:04:45 am]  * Found Image to transfer to 2 nodes
[11-07-18 7:04:45 am]  | Image Name: test
[11-07-18 7:04:46 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-07-18 7:04:46 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-1)
[11-07-18 7:04:46 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-07-18 7:04:47 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-07-18 7:04:47 am]  * All files synced for this item.
[11-07-18 7:04:47 am]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-07-18 7:04:47 am]   # test: No need to sync CentOS-7-x86_64-DVD-1804.iso (NODE-2)
[11-07-18 7:04:48 am]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-07-18 7:04:48 am]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-07-18 7:04:48 am]  * All files synced for this item.
[11-07-18 7:04:48 am]  | Sync finished - Resource id #975
[11-07-18 7:04:48 am]  | Sync finished - Resource id #993

[Sebastian Roth]

@jflippen Thanks heaps for testing and posting feedback! Log’s looking pretty good I reckon.

The first time around I tried just doing an in-place upgrade of the working 1.6 branch. This, however seemed to not work properly as it did not change anything after I swapped branches and installed on all three (stopping the service and rebooting on each VM, going from nodes to master).

This sounds strange. Cannot think of why it shouldn’t properly upgrade from working-1.6 to replication branch. On the other hand I have not tested myself yet. So it’s not impossible to cause an issue. Do you think it’s worth looking into? How would I replicate the issue?

During testing I noticed that the replication service won’t remove files that are on the node but not the master (from changing the files I was testing with on the master). Would it be possible to implement an if exist variable that checks an array of the file names in an image folder against the master? This would be handy for keeping the driver injection folder we use in the images folder synced properly too when removing deprecated models.

Got me! This is one last thing I hope to get implemented before the next release. This has been how replication is working for quite some time as it is not causing trouble in most cases. But you are absolutely right that we should implement that. Will do so!

[jflippen]

@Sebastian-Roth

This sounds strange. Cannot think of why it shouldn’t properly upgrade from working-1.6 to replication branch. On the other hand I have not tested myself yet. So it’s not impossible to cause an issue. Do you think it’s worth looking into? How would I replicate the issue?

Maybe, but I know it is generally considered best practice to back up the database and start from scratch when it comes to upgrading the FOG server (at least I think I remember reading that somewhere on the wiki or forums). Every time I tried “uninstalling” FOG it is never a clean uninstall. I generally haven’t had issues with doing an in-place update or switching branches, but like I said when i tried to do it for the replication branch going from working 1.6 it kept the new web UI and the log was spitting out the old way, making me believe that either the install skipped files that needed to be replaced or something else failed.

I do have a few other bugs I noticed, but I’ll have to make a new thread for those. I see that there is a bug section in the forum, but wasn’t sure if there was a page with a list of current known bugs or if I just need to search the forum for the bugs I have been experiencing. I don’t want to start a new thread if I don’t need to!

[Sebastian Roth]

@jflippen Usually just going forward with newer versions by updating the git repo and re-running the installer is working well. Going to working-1.6 and back is a bit different though.

I know there are many more parts of FOG that need fixing (uninstaller is just one of them) but we don’t have as many people helping out with FOG development right now. We hope to see people being interested in FOG to join in and help.

Nevertheless please report bugs as you find them and if they are not answered in the forums yet. But don’t be afraid to open new topics! Do a quick search and if you can’t find anything just post a new topic.

[Sebastian Roth]

@jflippen Ok, got around to implement that last bit to cleanup files on the storage nodes that might be there from earlier syncs. Simply run git pull to load the fixes and make sure you are still on the right branch via git branch -v. Then re-run the installer and see if that is doing the job properly. Waiting for feedback before I merge this into the working code. Thanks in advance.

[jflippen]

@Sebastian-Roth Sorry for the delay. I didn’t see your response until yesterday and didn’t get a chance to try until today. So, I did a git pull but the replication branch seems to have vanished! When I checked GitHub it is not present there either. Did you already merge it with the working branch? I am still willing to test it if you have a branch you would like me to use.

[jflippen]

@Sebastian-Roth Never mind. Answered my own question:

I’ll switch branches and do some testing.

[jflippen]

@Sebastian-Roth aaaand it looks like the updates are now removing files! Good job man! I really appreciate all the hard work you guys are putting into this project. Here is the log from my testing:

[11-14-18 5:42:22 pm] 
==================================
===        ====    =====      ====
===  =========  ==  ===   ==   ===
===  ========  ====  ==  ====  ===
===  ========  ====  ==  =========
===      ====  ====  ==  =========
===  ========  ====  ==  ===   ===
===  ========  ====  ==  ====  ===
===  =========  ==  ===   ==   ===
===  ==========    =====      ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[11-14-18 5:42:22 pm] Interface Ready with IP Address: 127.0.0.1
[11-14-18 5:42:22 pm] Interface Ready with IP Address: 127.0.1.1
[11-14-18 5:42:22 pm] Interface Ready with IP Address: 192.168.1.16
[11-14-18 5:42:22 pm] Interface Ready with IP Address: 192.168.122.1
[11-14-18 5:42:22 pm] Interface Ready with IP Address: FOG-MASTER
[11-14-18 5:42:22 pm]  * Starting ImageReplicator Service
[11-14-18 5:42:22 pm]  * Checking for new items every 600 seconds
[11-14-18 5:42:22 pm]  * Starting service loop
[11-14-18 5:42:22 pm]  * Starting Image Replication.
[11-14-18 5:42:22 pm]  * We are group ID: 1. We are group name: default
[11-14-18 5:42:22 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-14-18 5:42:22 pm]  * Attempting to perform Group -> Group image replication.
[11-14-18 5:42:22 pm]  | Replicating postdownloadscripts
[11-14-18 5:42:23 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:42:23 pm]  | File Name: postdownloadscripts
[11-14-18 5:42:23 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-1)
[11-14-18 5:42:23 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-1)
[11-14-18 5:42:23 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-14-18 5:42:23 pm]  * All files synced for this item.
[11-14-18 5:42:24 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-2)
[11-14-18 5:42:24 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-2)
[11-14-18 5:42:24 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-14-18 5:42:24 pm]  * All files synced for this item.
[11-14-18 5:42:24 pm]  | Replicating postinitscripts
[11-14-18 5:42:24 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:42:24 pm]  | File Name: dev/postinitscripts
[11-14-18 5:42:25 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-14-18 5:42:25 pm]  * All files synced for this item.
[11-14-18 5:42:25 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-14-18 5:42:25 pm]  * All files synced for this item.
[11-14-18 5:42:25 pm]  * Not syncing Image between groups
[11-14-18 5:42:25 pm]  | Image Name: test
[11-14-18 5:42:25 pm]  | There are no other members to sync to.
[11-14-18 5:42:25 pm]  * Attempting to perform Group -> Nodes image replication.
[11-14-18 5:42:26 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:42:26 pm]  | Image Name: test
[11-14-18 5:42:26 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-14-18 5:42:26 pm]   # test: File size mismatch - Fedora-Server-dvd-x86_64-28-1.1.iso: 4470079488 != 2903506944
[11-14-18 5:42:26 pm]   # test: Deleting remote file Fedora-Server-dvd-x86_64-28-1.1.iso
[11-14-18 5:42:26 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-14-18 5:42:26 pm]   # test: File does not exist on master node, deleting CentOS-7-x86_64-DVD-1804.iso on NODE-1
[11-14-18 5:42:26 pm]   # test: File does not exist on master node, deleting rufus-2.12.exe on NODE-1
[11-14-18 5:42:26 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogreplicator.test.transfer.NODE-1.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.17
[11-14-18 5:42:26 pm]  | Started sync for Image test - Resource id #663
[11-14-18 5:42:27 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-14-18 5:42:27 pm]   # test: File size mismatch - Fedora-Server-dvd-x86_64-28-1.1.iso: 4470079488 != 2903506944
[11-14-18 5:42:27 pm]   # test: Deleting remote file Fedora-Server-dvd-x86_64-28-1.1.iso
[11-14-18 5:42:27 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-14-18 5:42:27 pm]   # test: File does not exist on master node, deleting CentOS-7-x86_64-DVD-1804.iso on NODE-2
[11-14-18 5:42:27 pm]   # test: File does not exist on master node, deleting rufus-2.12.exe on NODE-2
[11-14-18 5:42:27 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogreplicator.test.transfer.NODE-2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/images/test" "/images/test"; exit' -u fog,[Protected] 192.168.1.18
[11-14-18 5:42:27 pm]  | Started sync for Image test - Resource id #690
[11-14-18 5:45:01 pm]  | Sync finished - Resource id #663
[11-14-18 5:45:01 pm]  | Sync finished - Resource id #690
[11-14-18 5:52:27 pm]  * Starting Image Replication.
[11-14-18 5:52:27 pm]  * We are group ID: 1. We are group name: default
[11-14-18 5:52:27 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-14-18 5:52:27 pm]  * Attempting to perform Group -> Group image replication.
[11-14-18 5:52:27 pm]  | Replicating postdownloadscripts
[11-14-18 5:52:27 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:52:27 pm]  | File Name: postdownloadscripts
[11-14-18 5:52:27 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-1)
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-1)
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-14-18 5:52:28 pm]  * All files synced for this item.
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-2)
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-2)
[11-14-18 5:52:28 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-14-18 5:52:28 pm]  * All files synced for this item.
[11-14-18 5:52:28 pm]  | Replicating postinitscripts
[11-14-18 5:52:29 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:52:29 pm]  | File Name: dev/postinitscripts
[11-14-18 5:52:29 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-14-18 5:52:29 pm]  * All files synced for this item.
[11-14-18 5:52:30 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-14-18 5:52:30 pm]  * All files synced for this item.
[11-14-18 5:52:30 pm]  * Not syncing Image between groups
[11-14-18 5:52:30 pm]  | Image Name: test
[11-14-18 5:52:30 pm]  | There are no other members to sync to.
[11-14-18 5:52:30 pm]  * Attempting to perform Group -> Nodes image replication.
[11-14-18 5:52:30 pm]  * Found Image to transfer to 2 nodes
[11-14-18 5:52:30 pm]  | Image Name: test
[11-14-18 5:52:31 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-14-18 5:52:31 pm]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-14-18 5:52:31 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-14-18 5:52:31 pm]  * All files synced for this item.
[11-14-18 5:52:32 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-14-18 5:52:32 pm]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-14-18 5:52:32 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-14-18 5:52:32 pm]  * All files synced for this item.
[11-14-18 6:02:27 pm]  * Starting Image Replication.
[11-14-18 6:02:27 pm]  * We are group ID: 1. We are group name: default
[11-14-18 6:02:27 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-14-18 6:02:27 pm]  * Attempting to perform Group -> Group image replication.
[11-14-18 6:02:27 pm]  | Replicating postdownloadscripts
[11-14-18 6:02:27 pm]  * Found Image to transfer to 2 nodes
[11-14-18 6:02:27 pm]  | File Name: postdownloadscripts
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-1)
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-1)
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-1)
[11-14-18 6:02:28 pm]  * All files synced for this item.
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.copydrivers (NODE-2)
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.custominstall (NODE-2)
[11-14-18 6:02:28 pm]   # postdownloadscripts: No need to sync fog.postdownload (NODE-2)
[11-14-18 6:02:28 pm]  * All files synced for this item.
[11-14-18 6:02:28 pm]  | Replicating postinitscripts
[11-14-18 6:02:29 pm]  * Found Image to transfer to 2 nodes
[11-14-18 6:02:29 pm]  | File Name: dev/postinitscripts
[11-14-18 6:02:29 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-1)
[11-14-18 6:02:29 pm]  * All files synced for this item.
[11-14-18 6:02:30 pm]   # dev/postinitscripts: No need to sync fog.postinit (NODE-2)
[11-14-18 6:02:30 pm]  * All files synced for this item.
[11-14-18 6:02:30 pm]  * Not syncing Image between groups
[11-14-18 6:02:30 pm]  | Image Name: test
[11-14-18 6:02:30 pm]  | There are no other members to sync to.
[11-14-18 6:02:30 pm]  * Attempting to perform Group -> Nodes image replication.
[11-14-18 6:02:30 pm]  * Found Image to transfer to 2 nodes
[11-14-18 6:02:30 pm]  | Image Name: test
[11-14-18 6:02:31 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-1)
[11-14-18 6:02:31 pm]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-1)
[11-14-18 6:02:31 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-1)
[11-14-18 6:02:31 pm]  * All files synced for this item.
[11-14-18 6:02:32 pm]   # test: No need to sync 2014-10-31 11.52.32.jpg (NODE-2)
[11-14-18 6:02:32 pm]   # test: No need to sync Fedora-Server-dvd-x86_64-28-1.1.iso (NODE-2)
[11-14-18 6:02:32 pm]   # test: No need to sync FileZilla_3.38.1_win64-setup.exe (NODE-2)
[11-14-18 6:02:32 pm]  * All files synced for this item.

[Sebastian Roth]

@jflippen Yeah, I did move on and merge it back into working as we are preparing for the next release coming fairly soon. Thanks again for testing. Seems fine from the logs.

[jflippen]

@Sebastian-Roth I decided to update our official FOG servers at work today to the newly updated working branch today and I am glad to say that the image replication is working now. However, It seems the snapin replication is having issues deleting files from the storage nodes. I am not sure if our previous manual rsync messed up the permissions.

I noticed on the FOG master that files are set to as such:

total 3035896
drwxrwxrwx. 3 fog  apache       4096 Nov 13 12:54 .
drwxr-xr-x. 5 root root           67 Nov 16 11:03 ..
-rwxrwxrwx  1 fog  apache  283844178 Jul  5 15:01 ActivInspire.zip
-rwxrwxrwx  1 fog  apache   78095360 Jun 20 14:03 A__Suite_for_Win_v2.5.2063.64.                msi
-rwxrwxrwx  1 fog  apache   82991148 Oct 25 09:00 lanschool.zip
-rwxrwxrwx  1 fog  apache  472627531 Jun 26 12:26 LoggerPro3_15.zip
-rwxrwxrwx  1 fog  apache    2244608 Jun 26 08:42 npGCPlugin2_2.4.1.3_allusers.m                si
-rwxrwxrwx  1 fog  apache  698146052 Jun 21 10:06 Office13_SP1.zip
-rwxrwxrwx  1 fog  apache 1360475209 Jun 28 08:17 Office2010.zip
-rwxrwxrwx  1 fog  apache        222 Jul 17 16:03 remove_OneDrive.bat
drwxrwxrwx. 3 fog  apache         54 Jan 18  2018 ssl
-rwxrwxrwx  1 fog  apache  130306048 Jun 26 09:30 Visualizer_win7_1.4.2.0.msi
-rwxrwxrwx  1 fog  apache       1209 Sep 25 13:49 wifiChrome.zip
-rwxrwxrwx  1 fog  apache         44 Sep 26 11:21 WinAct2.bat

Whereas on the node it is like this:

total 3364696
drwxr-xr-x. 3 fog  fog        4096 Nov 16 14:27 .
drwxr-xr-x. 5 root root         99 Nov 16 11:00 ..
-rwxr-xr-x  1 fog  fog   205078814 Jun 26 09:44 ActiveInspire.zip
-rwxr-xr-x  1 fog  fog   283844178 Nov 16 14:27 ActivInspire.zip
-rwxr-xr-x  1 fog  fog    78095360 Nov 16 14:27 A__Suite_for_Win_v2.5.2063.64.msi
-rwxr-xr-x  1 fog  fog   131607283 Oct 12 12:29 java_update.zip
-rwxr-xr-x  1 fog  fog    82991148 Nov 16 14:27 lanschool.zip
-rwxr-xr-x  1 fog  fog   472627531 Nov 16 14:27 LoggerPro3_15.zip
-rwxr-xr-x  1 fog  fog     2244608 Nov 16 14:27 npGCPlugin2_2.4.1.3_allusers.msi
-rwxr-xr-x  1 fog  fog   698146052 Nov 16 14:27 Office13_SP1.zip
-rwxr-xr-x  1 fog  fog  1360475209 Nov 16 14:27 Office2010.zip
-rwxr-xr-x  1 fog  fog         222 Nov 16 14:27 remove_OneDrive.bat
drwxr-xr-x. 3 fog  fog          54 Nov 16 14:27 ssl
-rwxr-xr-x  1 fog  fog   130306048 Nov 16 14:27 Visualizer_win7_1.4.2.0.msi
-rwxr-xr-x  1 fog  fog        1209 Nov 16 14:27 wifiChrome.zip
-rwxr-xr-x  1 fog  fog          44 Nov 16 14:27 WinAct2.bat

Is there something wrong with the permissions that is blocking the files from getting deleted? Below is the log from the snapin replication service:

[11-16-18 2:27:14 pm] 
==================================
=== ==== ===== ====
=== ========= == === == ===
=== ======== ==== == ==== ===
=== ======== ==== == =========
=== ==== ==== == =========
=== ======== ==== == === ===
=== ======== ==== == ==== ===
=== ========= == === == ===
=== ========== ===== ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[11-16-18 2:27:14 pm] Interface Ready with IP Address: 10.59.10.12
[11-16-18 2:27:14 pm] Interface Ready with IP Address: 127.0.0.1
[11-16-18 2:27:14 pm] Interface Ready with IP Address: 127.0.1.1
[11-16-18 2:27:14 pm] Interface Ready with IP Address: 192.168.122.1
[11-16-18 2:27:14 pm] Interface Ready with IP Address: 216.64.238.32
[11-16-18 2:27:14 pm] Interface Ready with IP Address: fog-master-tsd-tsdnet-turlock-k12-ca-us
[11-16-18 2:27:14 pm] Interface Ready with IP Address: natpool1.turlock.k12.ca.us
[11-16-18 2:27:14 pm] * Starting SnapinReplicator Service
[11-16-18 2:27:14 pm] * Checking for new items every 600 seconds
[11-16-18 2:27:14 pm] * Starting service loop
[11-16-18 2:27:14 pm] * Starting Snapin Replication.
[11-16-18 2:27:14 pm] * We are group ID: 1. We are group name: default
[11-16-18 2:27:14 pm] * We are node ID: 1. We are node name: DO-FOG-MASTER
[11-16-18 2:27:14 pm] * Attempting to perform Group -> Group snapin replication.
[11-16-18 2:27:15 pm] | Replicating ssl less private key
[11-16-18 2:27:15 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:15 pm] | File Name: ssl/fog.csr
[11-16-18 2:27:15 pm] # ssl/fog.csr: File does not exist optfogsnapinssslfog.csr (OC-FOG-NODE)
[11-16-18 2:27:15 pm] # ssl/fog.csr: File does not exist on master node, deleting fog.csr on OC-FOG-NODE
[11-16-18 2:27:15 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep..transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:15 pm] | Started sync for Snapin ssl/fog.csr - Resource id #537
[11-16-18 2:27:15 pm] # ssl/fog.csr: File does not exist optfogsnapinssslfog.csr (PH-FOG-NODE)
[11-16-18 2:27:15 pm] # ssl/fog.csr: File does not exist on master node, deleting fog.csr on PH-FOG-NODE
[11-16-18 2:27:15 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep..transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:15 pm] | Started sync for Snapin ssl/fog.csr - Resource id #547
[11-16-18 2:27:16 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:16 pm] | File Name: ssl/CA
[11-16-18 2:27:16 pm] * All files synced for this item.
[11-16-18 2:27:16 pm] * All files synced for this item.
[11-16-18 2:27:17 pm] * Not syncing Snapin between groups
[11-16-18 2:27:17 pm] | Snapin Name: ActivateWindows
[11-16-18 2:27:17 pm] | There are no other members to sync to.
[11-16-18 2:27:17 pm] * Not syncing Snapin between groups
[11-16-18 2:27:17 pm] | Snapin Name: ActivInspire
[11-16-18 2:27:17 pm] | There are no other members to sync to.
[11-16-18 2:27:17 pm] * Not syncing Snapin between groups
[11-16-18 2:27:17 pm] | Snapin Name: Gradecam Plugin Win10
[11-16-18 2:27:17 pm] | There are no other members to sync to.
[11-16-18 2:27:17 pm] * Not syncing Snapin between groups
[11-16-18 2:27:17 pm] | Snapin Name: IPEVO Visualizer
[11-16-18 2:27:17 pm] | There are no other members to sync to.
[11-16-18 2:27:17 pm] * Not syncing Snapin between groups
[11-16-18 2:27:17 pm] | Snapin Name: lanschool
[11-16-18 2:27:17 pm] | There are no other members to sync to.
[11-16-18 2:27:17 pm] * Not syncing Snapin between groups
[11-16-18 2:27:17 pm] | Snapin Name: Logger Pro 3_15
[11-16-18 2:27:17 pm] | There are no other members to sync to.
[11-16-18 2:27:18 pm] * Not syncing Snapin between groups
[11-16-18 2:27:18 pm] | Snapin Name: Office2010
[11-16-18 2:27:18 pm] | There are no other members to sync to.
[11-16-18 2:27:18 pm] * Not syncing Snapin between groups
[11-16-18 2:27:18 pm] | Snapin Name: Office2013
[11-16-18 2:27:18 pm] | There are no other members to sync to.
[11-16-18 2:27:18 pm] * Not syncing Snapin between groups
[11-16-18 2:27:18 pm] | Snapin Name: Remove_OneDrive
[11-16-18 2:27:18 pm] | There are no other members to sync to.
[11-16-18 2:27:18 pm] * Not syncing Snapin between groups
[11-16-18 2:27:18 pm] | Snapin Name: Sphere 2
[11-16-18 2:27:18 pm] | There are no other members to sync to.
[11-16-18 2:27:18 pm] * Not syncing Snapin between groups
[11-16-18 2:27:18 pm] | Snapin Name: wifiChrome
[11-16-18 2:27:18 pm] | There are no other members to sync to.
[11-16-18 2:27:18 pm] * Attempting to perform Group -> Nodes snapin replication.
[11-16-18 2:27:19 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:19 pm] | Snapin Name: ActivateWindows
[11-16-18 2:27:19 pm] # ActivateWindows: File does not exist optfogsnapinsWinAct2.bat (OC-FOG-NODE)
[11-16-18 2:27:19 pm] # ActivateWindows: File does not exist on master node, deleting WinAct2.bat on OC-FOG-NODE
[11-16-18 2:27:19 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.ActivateWindows.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "WinAct2.bat" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:19 pm] | Started sync for Snapin ActivateWindows - Resource id #776
[11-16-18 2:27:19 pm] # ActivateWindows: File does not exist optfogsnapinsWinAct2.bat (PH-FOG-NODE)
[11-16-18 2:27:19 pm] # ActivateWindows: File does not exist on master node, deleting WinAct2.bat on PH-FOG-NODE
[11-16-18 2:27:19 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.ActivateWindows.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "WinAct2.bat" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:19 pm] | Started sync for Snapin ActivateWindows - Resource id #786
[11-16-18 2:27:20 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:20 pm] | Snapin Name: ActivInspire
[11-16-18 2:27:20 pm] # ActivInspire: File does not exist optfogsnapinsActivInspire.zip (OC-FOG-NODE)
[11-16-18 2:27:20 pm] # ActivInspire: File does not exist on master node, deleting ActivInspire.zip on OC-FOG-NODE
[11-16-18 2:27:20 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.ActivInspire.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "ActivInspire.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:20 pm] | Started sync for Snapin ActivInspire - Resource id #819
[11-16-18 2:27:20 pm] # ActivInspire: File does not exist optfogsnapinsActivInspire.zip (PH-FOG-NODE)
[11-16-18 2:27:20 pm] # ActivInspire: File does not exist on master node, deleting ActivInspire.zip on PH-FOG-NODE
[11-16-18 2:27:20 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.ActivInspire.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "ActivInspire.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:20 pm] | Started sync for Snapin ActivInspire - Resource id #829
[11-16-18 2:27:23 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:23 pm] | Snapin Name: Gradecam Plugin Win10
[11-16-18 2:27:23 pm] # Gradecam Plugin Win10: File does not exist optfogsnapinsnpGCPlugin2_2.4.1.3_allusers.msi (OC-FOG-NODE)
[11-16-18 2:27:23 pm] # Gradecam Plugin Win10: File does not exist on master node, deleting npGCPlugin2_2.4.1.3_allusers.msi on OC-FOG-NODE
[11-16-18 2:27:23 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Gradecam Plugin Win10.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "npGCPlugin2_2.4.1.3_allusers.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:23 pm] | Started sync for Snapin Gradecam Plugin Win10 - Resource id #862
[11-16-18 2:27:24 pm] # Gradecam Plugin Win10: File does not exist optfogsnapinsnpGCPlugin2_2.4.1.3_allusers.msi (PH-FOG-NODE)
[11-16-18 2:27:24 pm] # Gradecam Plugin Win10: File does not exist on master node, deleting npGCPlugin2_2.4.1.3_allusers.msi on PH-FOG-NODE
[11-16-18 2:27:24 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Gradecam Plugin Win10.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "npGCPlugin2_2.4.1.3_allusers.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:24 pm] | Started sync for Snapin Gradecam Plugin Win10 - Resource id #872
[11-16-18 2:27:24 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:24 pm] | Snapin Name: IPEVO Visualizer
[11-16-18 2:27:24 pm] # IPEVO Visualizer: File does not exist optfogsnapinsVisualizer_win7_1.4.2.0.msi (OC-FOG-NODE)
[11-16-18 2:27:24 pm] # IPEVO Visualizer: File does not exist on master node, deleting Visualizer_win7_1.4.2.0.msi on OC-FOG-NODE
[11-16-18 2:27:24 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.IPEVO Visualizer.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "Visualizer_win7_1.4.2.0.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:24 pm] | Started sync for Snapin IPEVO Visualizer - Resource id #905
[11-16-18 2:27:25 pm] # IPEVO Visualizer: File does not exist optfogsnapinsVisualizer_win7_1.4.2.0.msi (PH-FOG-NODE)
[11-16-18 2:27:25 pm] # IPEVO Visualizer: File does not exist on master node, deleting Visualizer_win7_1.4.2.0.msi on PH-FOG-NODE
[11-16-18 2:27:25 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.IPEVO Visualizer.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "Visualizer_win7_1.4.2.0.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:25 pm] | Started sync for Snapin IPEVO Visualizer - Resource id #915
[11-16-18 2:27:26 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:26 pm] | Snapin Name: lanschool
[11-16-18 2:27:26 pm] # lanschool: File does not exist optfogsnapinslanschool.zip (OC-FOG-NODE)
[11-16-18 2:27:26 pm] # lanschool: File does not exist on master node, deleting lanschool.zip on OC-FOG-NODE
[11-16-18 2:27:26 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.lanschool.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "lanschool.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:26 pm] | Started sync for Snapin lanschool - Resource id #948
[11-16-18 2:27:27 pm] # lanschool: File does not exist optfogsnapinslanschool.zip (PH-FOG-NODE)
[11-16-18 2:27:27 pm] # lanschool: File does not exist on master node, deleting lanschool.zip on PH-FOG-NODE
[11-16-18 2:27:27 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.lanschool.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "lanschool.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:27 pm] | Started sync for Snapin lanschool - Resource id #958
[11-16-18 2:27:28 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:28 pm] | Snapin Name: Logger Pro 3_15
[11-16-18 2:27:28 pm] # Logger Pro 3_15: File does not exist optfogsnapinsLoggerPro3_15.zip (OC-FOG-NODE)
[11-16-18 2:27:28 pm] # Logger Pro 3_15: File does not exist on master node, deleting LoggerPro3_15.zip on OC-FOG-NODE
[11-16-18 2:27:28 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Logger Pro 3_15.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "LoggerPro3_15.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:28 pm] | Started sync for Snapin Logger Pro 3_15 - Resource id #991
[11-16-18 2:27:28 pm] # Logger Pro 3_15: File does not exist optfogsnapinsLoggerPro3_15.zip (PH-FOG-NODE)
[11-16-18 2:27:28 pm] # Logger Pro 3_15: File does not exist on master node, deleting LoggerPro3_15.zip on PH-FOG-NODE
[11-16-18 2:27:28 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Logger Pro 3_15.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "LoggerPro3_15.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:28 pm] | Started sync for Snapin Logger Pro 3_15 - Resource id #1001
[11-16-18 2:27:32 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:32 pm] | Snapin Name: Office2010
[11-16-18 2:27:33 pm] # Office2010: File does not exist optfogsnapinsOffice2010.zip (OC-FOG-NODE)
[11-16-18 2:27:33 pm] # Office2010: File does not exist on master node, deleting Office2010.zip on OC-FOG-NODE
[11-16-18 2:27:33 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Office2010.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "Office2010.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:33 pm] | Started sync for Snapin Office2010 - Resource id #1034
[11-16-18 2:27:33 pm] # Office2010: File does not exist optfogsnapinsOffice2010.zip (PH-FOG-NODE)
[11-16-18 2:27:33 pm] # Office2010: File does not exist on master node, deleting Office2010.zip on PH-FOG-NODE
[11-16-18 2:27:33 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Office2010.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "Office2010.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:33 pm] | Started sync for Snapin Office2010 - Resource id #1044
[11-16-18 2:27:45 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:45 pm] | Snapin Name: Office2013
[11-16-18 2:27:45 pm] # Office2013: File does not exist optfogsnapinsOffice13_SP1.zip (OC-FOG-NODE)
[11-16-18 2:27:45 pm] # Office2013: File does not exist on master node, deleting Office13_SP1.zip on OC-FOG-NODE
[11-16-18 2:27:46 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Office2013.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "Office13_SP1.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:46 pm] | Started sync for Snapin Office2013 - Resource id #1077
[11-16-18 2:27:46 pm] # Office2013: File does not exist optfogsnapinsOffice13_SP1.zip (PH-FOG-NODE)
[11-16-18 2:27:46 pm] # Office2013: File does not exist on master node, deleting Office13_SP1.zip on PH-FOG-NODE
[11-16-18 2:27:46 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Office2013.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "Office13_SP1.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:46 pm] | Started sync for Snapin Office2013 - Resource id #1087
[11-16-18 2:27:52 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:52 pm] | Snapin Name: Remove_OneDrive
[11-16-18 2:27:52 pm] # Remove_OneDrive: File does not exist optfogsnapinsremove_OneDrive.bat (OC-FOG-NODE)
[11-16-18 2:27:52 pm] # Remove_OneDrive: File does not exist on master node, deleting remove_OneDrive.bat on OC-FOG-NODE
[11-16-18 2:27:52 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Remove_OneDrive.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "remove_OneDrive.bat" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:52 pm] | Started sync for Snapin Remove_OneDrive - Resource id #1120
[11-16-18 2:27:53 pm] # Remove_OneDrive: File does not exist optfogsnapinsremove_OneDrive.bat (PH-FOG-NODE)
[11-16-18 2:27:53 pm] # Remove_OneDrive: File does not exist on master node, deleting remove_OneDrive.bat on PH-FOG-NODE
[11-16-18 2:27:53 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Remove_OneDrive.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "remove_OneDrive.bat" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:53 pm] | Started sync for Snapin Remove_OneDrive - Resource id #1130
[11-16-18 2:27:53 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:53 pm] | Snapin Name: Sphere 2
[11-16-18 2:27:53 pm] # Sphere 2: File does not exist optfogsnapinsA__Suite_for_Win_v2.5.2063.64.msi (OC-FOG-NODE)
[11-16-18 2:27:53 pm] # Sphere 2: File does not exist on master node, deleting A__Suite_for_Win_v2.5.2063.64.msi on OC-FOG-NODE
[11-16-18 2:27:53 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Sphere 2.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "A__Suite_for_Win_v2.5.2063.64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:53 pm] | Started sync for Snapin Sphere 2 - Resource id #1163
[11-16-18 2:27:54 pm] # Sphere 2: File does not exist optfogsnapinsA__Suite_for_Win_v2.5.2063.64.msi (PH-FOG-NODE)
[11-16-18 2:27:54 pm] # Sphere 2: File does not exist on master node, deleting A__Suite_for_Win_v2.5.2063.64.msi on PH-FOG-NODE
[11-16-18 2:27:54 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Sphere 2.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "A__Suite_for_Win_v2.5.2063.64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:54 pm] | Started sync for Snapin Sphere 2 - Resource id #1173
[11-16-18 2:27:55 pm] * Found Snapin to transfer to 2 nodes
[11-16-18 2:27:55 pm] | Snapin Name: wifiChrome
[11-16-18 2:27:55 pm] # wifiChrome: File does not exist optfogsnapinswifiChrome.zip (OC-FOG-NODE)
[11-16-18 2:27:55 pm] # wifiChrome: File does not exist on master node, deleting wifiChrome.zip on OC-FOG-NODE
[11-16-18 2:27:55 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.wifiChrome.transfer.OC-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "wifiChrome.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.59.181.12
[11-16-18 2:27:55 pm] | Started sync for Snapin wifiChrome - Resource id #1206
[11-16-18 2:27:55 pm] # wifiChrome: File does not exist optfogsnapinswifiChrome.zip (PH-FOG-NODE)
[11-16-18 2:27:55 pm] # wifiChrome: File does not exist on master node, deleting wifiChrome.zip on PH-FOG-NODE
[11-16-18 2:27:55 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.wifiChrome.transfer.PH-FOG-NODE.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "wifiChrome.zip" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.60.219.12
[11-16-18 2:27:55 pm] | Started sync for Snapin wifiChrome - Resource id #1216```

[Sebastian Roth]

@jflippen Thanks again for testing and reporting. I definitely messed something up with snapin replication. Whooops. Will fix that on the weekend.

[jflippen]

@Sebastian-Roth Okay, sounds good. I will test any patches once you are able to release them. Happy holidays!

[Sebastian Roth]

@jflippen Ok, snapin replication should be fixed now. This time I pushed the changes to dev-branch.

cd fogproject
git pull
git checkout dev-branch
cd bin
./installfog.sh

[jflippen]

@Sebastian-Roth okay, so good news and bad news. First, the good news. The good news is that the snapin replication service will now delete and replace a snapin file that has been updated.

Now for the bad news, possibly… depending on how you intend the snapin service to function.

1. If I upload a different filename for a snapin, the old file associated with the snapin does not get removed from either the master or the storage nodes. This can be an issue for MSI files that update their file name with each release (e.g. Java updates). I have found sometimes I may accidentally have the syntax wrong as well and have to re-upload a file. However, this leaves me manually going into the snapins folder to cleanup old files. Not a deal breaker, but I felt this is something that should be noted.

2. The snapin replication service does not remove files in the snapin folder that are not associated with snapins. This kind of ties in with the previous issue, and may not be a real-world example, but I did notice that if I put a file in the snapin folder on the storage node folder with the proper permissions, the service does not even check for rogue / old files. I know for the images replication you had it clean up files not on the master, so I was checking for that functionality in the snapin replication as well.

3. Again, may not be a real-world scenario, but I tried setting a new snapin to an existing file that hadn’t gotten erased (from uploading a different filename to a previous snapin) and it changed the file to a size of 0kb. This empty file then got replicated…so, users beware. Do not create a snapin and then attach to an existing file in the snapin folder.

Here is the changes I noticed in the files while playing around:

Here is my snapin replication log from today, where you can see the empty file as well as the lack of detecting the 7zip MSI:

[11-24-18 8:05:59 pm] 
==================================
===        ====    =====      ====
===  =========  ==  ===   ==   ===
===  ========  ====  ==  ====  ===
===  ========  ====  ==  =========
===      ====  ====  ==  =========
===  ========  ====  ==  ===   ===
===  ========  ====  ==  ====  ===
===  =========  ==  ===   ==   ===
===  ==========    =====      ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[11-24-18 8:05:59 pm] Interface Ready with IP Address: 127.0.0.1
[11-24-18 8:05:59 pm] Interface Ready with IP Address: 127.0.1.1
[11-24-18 8:05:59 pm] Interface Ready with IP Address: 192.168.1.16
[11-24-18 8:05:59 pm] Interface Ready with IP Address: 192.168.122.1
[11-24-18 8:05:59 pm] Interface Ready with IP Address: FOG-MASTER
[11-24-18 8:05:59 pm]  * Starting SnapinReplicator Service
[11-24-18 8:05:59 pm]  * Checking for new items every 600 seconds
[11-24-18 8:05:59 pm]  * Starting service loop
[11-24-18 8:05:59 pm]  * Starting Snapin Replication.
[11-24-18 8:05:59 pm]  * We are group ID: 1. We are group name: default
[11-24-18 8:05:59 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-24-18 8:05:59 pm]  * Attempting to perform Group -> Group snapin replication.
[11-24-18 8:05:59 pm]  | Replicating ssl less private key
[11-24-18 8:06:00 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:06:00 pm]  | File Name: ssl/fog.csr
[11-24-18 8:06:00 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-1)
[11-24-18 8:06:00 pm]  * All files synced for this item.
[11-24-18 8:06:01 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-2)
[11-24-18 8:06:01 pm]  * All files synced for this item.
[11-24-18 8:06:01 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:06:01 pm]  | File Name: ssl/CA
[11-24-18 8:06:01 pm]  * All files synced for this item.
[11-24-18 8:06:02 pm]  * All files synced for this item.
[11-24-18 8:06:02 pm]  * Not syncing Snapin between groups
[11-24-18 8:06:02 pm]  | Snapin Name: Chrome x64
[11-24-18 8:06:02 pm]  | There are no other members to sync to.
[11-24-18 8:06:02 pm]  * Attempting to perform Group -> Nodes snapin replication.
[11-24-18 8:06:02 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:06:02 pm]  | Snapin Name: Chrome x64
[11-24-18 8:06:03 pm]   # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-1)
[11-24-18 8:06:03 pm]  * All files synced for this item.
[11-24-18 8:06:04 pm]   # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-2)
[11-24-18 8:06:04 pm]  * All files synced for this item.
[11-24-18 8:16:04 pm]  * Starting Snapin Replication.
[11-24-18 8:16:04 pm]  * We are group ID: 1. We are group name: default
[11-24-18 8:16:04 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-24-18 8:16:04 pm]  * Attempting to perform Group -> Group snapin replication.
[11-24-18 8:16:04 pm]  | Replicating ssl less private key
[11-24-18 8:16:04 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:16:04 pm]  | File Name: ssl/fog.csr
[11-24-18 8:16:05 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-1)
[11-24-18 8:16:05 pm]  * All files synced for this item.
[11-24-18 8:16:06 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-2)
[11-24-18 8:16:06 pm]  * All files synced for this item.
[11-24-18 8:16:06 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:16:06 pm]  | File Name: ssl/CA
[11-24-18 8:16:06 pm]  * All files synced for this item.
[11-24-18 8:16:07 pm]  * All files synced for this item.
[11-24-18 8:16:07 pm]  * Not syncing Snapin between groups
[11-24-18 8:16:07 pm]  | Snapin Name: Chrome x64
[11-24-18 8:16:07 pm]  | There are no other members to sync to.
[11-24-18 8:16:07 pm]  * Attempting to perform Group -> Nodes snapin replication.
[11-24-18 8:16:08 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:16:08 pm]  | Snapin Name: Chrome x64
[11-24-18 8:16:08 pm]   # Chrome x64: File does not exist googlechromebetastandaloneenterprise64.msi (NODE-1)
[11-24-18 8:16:08 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome x64.transfer.NODE-1.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.17
[11-24-18 8:16:08 pm]  | Started sync for Snapin Chrome x64 - Resource id #761
[11-24-18 8:16:08 pm]   # Chrome x64: File does not exist googlechromebetastandaloneenterprise64.msi (NODE-2)
[11-24-18 8:16:08 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome x64.transfer.NODE-2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.18
[11-24-18 8:16:08 pm]  | Started sync for Snapin Chrome x64 - Resource id #770
[11-24-18 8:16:09 pm]  | Sync finished - Resource id #761
[11-24-18 8:16:09 pm]  | Sync finished - Resource id #770
[11-24-18 8:26:04 pm]  * Starting Snapin Replication.
[11-24-18 8:26:04 pm]  * We are group ID: 1. We are group name: default
[11-24-18 8:26:04 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-24-18 8:26:04 pm]  * Attempting to perform Group -> Group snapin replication.
[11-24-18 8:26:04 pm]  | Replicating ssl less private key
[11-24-18 8:26:04 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:26:04 pm]  | File Name: ssl/fog.csr
[11-24-18 8:26:05 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-1)
[11-24-18 8:26:05 pm]  * All files synced for this item.
[11-24-18 8:26:05 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-2)
[11-24-18 8:26:05 pm]  * All files synced for this item.
[11-24-18 8:26:06 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:26:06 pm]  | File Name: ssl/CA
[11-24-18 8:26:06 pm]  * All files synced for this item.
[11-24-18 8:26:06 pm]  * All files synced for this item.
[11-24-18 8:26:07 pm]  * Not syncing Snapin between groups
[11-24-18 8:26:07 pm]  | Snapin Name: Chrome x64
[11-24-18 8:26:07 pm]  | There are no other members to sync to.
[11-24-18 8:26:07 pm]  * Attempting to perform Group -> Nodes snapin replication.
[11-24-18 8:26:07 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:26:07 pm]  | Snapin Name: Chrome x64
[11-24-18 8:26:08 pm]   # Chrome x64: No need to sync googlechromebetastandaloneenterprise64.msi (NODE-1)
[11-24-18 8:26:08 pm]  * All files synced for this item.
[11-24-18 8:26:09 pm]   # Chrome x64: No need to sync googlechromebetastandaloneenterprise64.msi (NODE-2)
[11-24-18 8:26:09 pm]  * All files synced for this item.
[11-24-18 8:36:04 pm]  * Starting Snapin Replication.
[11-24-18 8:36:04 pm]  * We are group ID: 1. We are group name: default
[11-24-18 8:36:04 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-24-18 8:36:04 pm]  * Attempting to perform Group -> Group snapin replication.
[11-24-18 8:36:04 pm]  | Replicating ssl less private key
[11-24-18 8:36:04 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:36:04 pm]  | File Name: ssl/fog.csr
[11-24-18 8:36:05 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-1)
[11-24-18 8:36:05 pm]  * All files synced for this item.
[11-24-18 8:36:06 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-2)
[11-24-18 8:36:06 pm]  * All files synced for this item.
[11-24-18 8:36:06 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:36:06 pm]  | File Name: ssl/CA
[11-24-18 8:36:06 pm]  * All files synced for this item.
[11-24-18 8:36:07 pm]  * All files synced for this item.
[11-24-18 8:36:07 pm]  * Not syncing Snapin between groups
[11-24-18 8:36:07 pm]  | Snapin Name: Chrome Beta
[11-24-18 8:36:07 pm]  | There are no other members to sync to.
[11-24-18 8:36:07 pm]  * Not syncing Snapin between groups
[11-24-18 8:36:07 pm]  | Snapin Name: Chrome x64
[11-24-18 8:36:07 pm]  | There are no other members to sync to.
[11-24-18 8:36:07 pm]  * Attempting to perform Group -> Nodes snapin replication.
[11-24-18 8:36:08 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:36:08 pm]  | Snapin Name: Chrome Beta
[11-24-18 8:36:08 pm]   # Chrome Beta: File size mismatch - googlechromebetastandaloneenterprise64.msi: 0 != 56467664
[11-24-18 8:36:08 pm]   # Chrome Beta: Deleting remote file googlechromebetastandaloneenterprise64.msi
[11-24-18 8:36:08 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome Beta.transfer.NODE-1.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.17
[11-24-18 8:36:08 pm]  | Started sync for Snapin Chrome Beta - Resource id #1058
[11-24-18 8:36:09 pm]   # Chrome Beta: File size mismatch - googlechromebetastandaloneenterprise64.msi: 0 != 56467664
[11-24-18 8:36:09 pm]   # Chrome Beta: Deleting remote file googlechromebetastandaloneenterprise64.msi
[11-24-18 8:36:09 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome Beta.transfer.NODE-2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.18
[11-24-18 8:36:09 pm]  | Started sync for Snapin Chrome Beta - Resource id #1071
[11-24-18 8:36:09 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:36:09 pm]  | Snapin Name: Chrome x64
[11-24-18 8:36:10 pm]   # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-1)
[11-24-18 8:36:10 pm]  * All files synced for this item.
[11-24-18 8:36:11 pm]   # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-2)
[11-24-18 8:36:11 pm]  * All files synced for this item.
[11-24-18 8:36:11 pm]  | Sync finished - Resource id #1058
[11-24-18 8:36:11 pm]  | Sync finished - Resource id #1071
[11-24-18 8:46:04 pm]  * Starting Snapin Replication.
[11-24-18 8:46:04 pm]  * We are group ID: 1. We are group name: default
[11-24-18 8:46:04 pm]  * We are node ID: 1. We are node name: DefaultMember
[11-24-18 8:46:04 pm]  * Attempting to perform Group -> Group snapin replication.
[11-24-18 8:46:04 pm]  | Replicating ssl less private key
[11-24-18 8:46:04 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:46:04 pm]  | File Name: ssl/fog.csr
[11-24-18 8:46:05 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-1)
[11-24-18 8:46:05 pm]  * All files synced for this item.
[11-24-18 8:46:05 pm]   # ssl/fog.csr: No need to sync fog.csr (NODE-2)
[11-24-18 8:46:05 pm]  * All files synced for this item.
[11-24-18 8:46:06 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:46:06 pm]  | File Name: ssl/CA
[11-24-18 8:46:06 pm]  * All files synced for this item.
[11-24-18 8:46:06 pm]  * All files synced for this item.
[11-24-18 8:46:07 pm]  * Not syncing Snapin between groups
[11-24-18 8:46:07 pm]  | Snapin Name: Chrome Beta
[11-24-18 8:46:07 pm]  | There are no other members to sync to.
[11-24-18 8:46:07 pm]  * Not syncing Snapin between groups
[11-24-18 8:46:07 pm]  | Snapin Name: Chrome x64
[11-24-18 8:46:07 pm]  | There are no other members to sync to.
[11-24-18 8:46:07 pm]  * Attempting to perform Group -> Nodes snapin replication.
[11-24-18 8:46:07 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:46:07 pm]  | Snapin Name: Chrome Beta
[11-24-18 8:46:08 pm]   # Chrome Beta: File size mismatch - googlechromebetastandaloneenterprise64.msi: 56467664 != 0
[11-24-18 8:46:08 pm]   # Chrome Beta: Deleting remote file googlechromebetastandaloneenterprise64.msi
[11-24-18 8:46:08 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome Beta.transfer.NODE-1.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.17
[11-24-18 8:46:08 pm]  | Started sync for Snapin Chrome Beta - Resource id #1261
[11-24-18 8:46:08 pm]   # Chrome Beta: File size mismatch - googlechromebetastandaloneenterprise64.msi: 56467664 != 0
[11-24-18 8:46:08 pm]   # Chrome Beta: Deleting remote file googlechromebetastandaloneenterprise64.msi
[11-24-18 8:46:08 pm]  | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome Beta.transfer.NODE-2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.18
[11-24-18 8:46:08 pm]  | Started sync for Snapin Chrome Beta - Resource id #1274
[11-24-18 8:46:09 pm]  * Found Snapin to transfer to 2 nodes
[11-24-18 8:46:09 pm]  | Snapin Name: Chrome x64
[11-24-18 8:46:10 pm]   # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-1)
[11-24-18 8:46:10 pm]  * All files synced for this item.
[11-24-18 8:46:10 pm]   # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-2)
[11-24-18 8:46:10 pm]  * All files synced for this item.
[11-24-18 8:46:10 pm]  | Sync finished - Resource id #1261
[11-24-18 8:46:10 pm]  | Sync finished - Resource id #1274

4. This last issue is for both images and snapins, but is more based on the something I noticed in the web UI, so I’m not sure which dev to report this to as a possible bug, or if it’s more of a feature request…but if you go to delete one or more snapins / images from the list of all snapins / all images, it will not remove the files associated with whatever you delete. In order to do so, you must enter each image / snapin individually, go to the delete tab, and check the box to delete files when deleting. It would be nice to have on the all images / snapins pages for mass changes or just trying to explain to my team that only one path of deleting gives expected results.

Please put this option:

Here:

Sorry if it seems like I am nitpicking, I am just trying to be thorough in my testing and ran tests based off of issues I have come across myself by accident. I am sure there are users who will never encounter the scenarios I have provided, but I am trying to cover any possible future headaches. Yay QA testing…

[Sebastian Roth]

@jflippen Thanks heaps for testing and your lengthy report, highly appreciated! Now about your questions/findings:

1. Good point. So essentially the snapin binaries pile up over time. On the other hand I am wondering if we should simply remove any unknown files from the snapin directory? What if we delete a file which someone put there by intention and didn’t want us to delete it? The best way I find would be to ask the user if we should delete the old binary when editing a snapin takes place and a different filename is used. This way people can decide if they want to keep the old binary on the server (might use it later) or not. What do you think? I created a github issue to keep track of this.

2. As you said, this ties in very closely with 1. and I am tending to be conservative here and not just delete files we don’t know about. You are right that we sync and delete files within /images/<IMAGENAME>/... sub directories. It’s too much to keep track of all individual files in there. But if you simply create a directory /images/test123/ and put files into that, replication won’t touch it! Same with files in the snapin directory which are not associated anymore.

3. Oh yeah, not good. Found and fixed (in dev-branch). But seems like this bug has been there for over a year now. We definitely need more people to test FOG as well as developers to fix stuff.

4. Yeah would be more like a feature request. May I ask you to post this as a new topic in the forums (section feature requests) so it’s not lost here. But I won’t promise we’ll implement it. As already mentioned we are careful with deleting files via web interface. Especially this kind of “mass deletion”. Pleas post and we’ll discuss this with all the others.

[jflippen]

@Sebastian-Roth

1. Your idea sounds like a good compromise. Something like delete old file on new upload checkbox that is checked by default maybe?

2. okay, that makes sense. I forgot that the images folder only syncs folders tied to an image in the table.

3. I tried your update and that fixed the file getting set to 0kb. I test when I can, as FOG has a lot of potential in my eyes and I know most if not all of you are doing this in your spare time, which means a lot. I’m hoping someday I’ll have time to learn PHP and contribute more to the project, but that’s a battle for another day.

4. Posted in feature request and tagged you and this post for reference.