Replication problems 1.5.4 - always copying
-
@Sebastian-Roth Okay, sounds good. I will test any patches once you are able to release them. Happy holidays!
-
@jflippen Ok, snapin replication should be fixed now. This time I pushed the changes to
dev-branch.cd fogproject git pull git checkout dev-branch cd bin ./installfog.sh -
@Sebastian-Roth okay, so good news and bad news. First, the good news. The good news is that the snapin replication service will now delete and replace a snapin file that has been updated.
Now for the bad news, possibly… depending on how you intend the snapin service to function.
-
If I upload a different filename for a snapin, the old file associated with the snapin does not get removed from either the master or the storage nodes. This can be an issue for MSI files that update their file name with each release (e.g. Java updates). I have found sometimes I may accidentally have the syntax wrong as well and have to re-upload a file. However, this leaves me manually going into the snapins folder to cleanup old files. Not a deal breaker, but I felt this is something that should be noted.
-
The snapin replication service does not remove files in the snapin folder that are not associated with snapins. This kind of ties in with the previous issue, and may not be a real-world example, but I did notice that if I put a file in the snapin folder on the storage node folder with the proper permissions, the service does not even check for rogue / old files. I know for the images replication you had it clean up files not on the master, so I was checking for that functionality in the snapin replication as well.
-
Again, may not be a real-world scenario, but I tried setting a new snapin to an existing file that hadn’t gotten erased (from uploading a different filename to a previous snapin) and it changed the file to a size of 0kb. This empty file then got replicated…so, users beware. Do not create a snapin and then attach to an existing file in the snapin folder.
Here is the changes I noticed in the files while playing around:
Here is my snapin replication log from today, where you can see the empty file as well as the lack of detecting the 7zip MSI:
[11-24-18 8:05:59 pm] ================================== === ==== ===== ==== === ========= == === == === === ======== ==== == ==== === === ======== ==== == ========= === ==== ==== == ========= === ======== ==== == === === === ======== ==== == ==== === === ========= == === == === === ========== ===== ==== ================================== ===== Free Opensource Ghost ====== ================================== ============ Credits ============= = https://fogproject.org/Credits = ================================== == Released under GPL Version 3 == ================================== [11-24-18 8:05:59 pm] Interface Ready with IP Address: 127.0.0.1 [11-24-18 8:05:59 pm] Interface Ready with IP Address: 127.0.1.1 [11-24-18 8:05:59 pm] Interface Ready with IP Address: 192.168.1.16 [11-24-18 8:05:59 pm] Interface Ready with IP Address: 192.168.122.1 [11-24-18 8:05:59 pm] Interface Ready with IP Address: FOG-MASTER [11-24-18 8:05:59 pm] * Starting SnapinReplicator Service [11-24-18 8:05:59 pm] * Checking for new items every 600 seconds [11-24-18 8:05:59 pm] * Starting service loop [11-24-18 8:05:59 pm] * Starting Snapin Replication. [11-24-18 8:05:59 pm] * We are group ID: 1. We are group name: default [11-24-18 8:05:59 pm] * We are node ID: 1. We are node name: DefaultMember [11-24-18 8:05:59 pm] * Attempting to perform Group -> Group snapin replication. [11-24-18 8:05:59 pm] | Replicating ssl less private key [11-24-18 8:06:00 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:06:00 pm] | File Name: ssl/fog.csr [11-24-18 8:06:00 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-1) [11-24-18 8:06:00 pm] * All files synced for this item. [11-24-18 8:06:01 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-2) [11-24-18 8:06:01 pm] * All files synced for this item. [11-24-18 8:06:01 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:06:01 pm] | File Name: ssl/CA [11-24-18 8:06:01 pm] * All files synced for this item. [11-24-18 8:06:02 pm] * All files synced for this item. [11-24-18 8:06:02 pm] * Not syncing Snapin between groups [11-24-18 8:06:02 pm] | Snapin Name: Chrome x64 [11-24-18 8:06:02 pm] | There are no other members to sync to. [11-24-18 8:06:02 pm] * Attempting to perform Group -> Nodes snapin replication. [11-24-18 8:06:02 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:06:02 pm] | Snapin Name: Chrome x64 [11-24-18 8:06:03 pm] # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-1) [11-24-18 8:06:03 pm] * All files synced for this item. [11-24-18 8:06:04 pm] # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-2) [11-24-18 8:06:04 pm] * All files synced for this item. [11-24-18 8:16:04 pm] * Starting Snapin Replication. [11-24-18 8:16:04 pm] * We are group ID: 1. We are group name: default [11-24-18 8:16:04 pm] * We are node ID: 1. We are node name: DefaultMember [11-24-18 8:16:04 pm] * Attempting to perform Group -> Group snapin replication. [11-24-18 8:16:04 pm] | Replicating ssl less private key [11-24-18 8:16:04 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:16:04 pm] | File Name: ssl/fog.csr [11-24-18 8:16:05 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-1) [11-24-18 8:16:05 pm] * All files synced for this item. [11-24-18 8:16:06 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-2) [11-24-18 8:16:06 pm] * All files synced for this item. [11-24-18 8:16:06 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:16:06 pm] | File Name: ssl/CA [11-24-18 8:16:06 pm] * All files synced for this item. [11-24-18 8:16:07 pm] * All files synced for this item. [11-24-18 8:16:07 pm] * Not syncing Snapin between groups [11-24-18 8:16:07 pm] | Snapin Name: Chrome x64 [11-24-18 8:16:07 pm] | There are no other members to sync to. [11-24-18 8:16:07 pm] * Attempting to perform Group -> Nodes snapin replication. [11-24-18 8:16:08 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:16:08 pm] | Snapin Name: Chrome x64 [11-24-18 8:16:08 pm] # Chrome x64: File does not exist googlechromebetastandaloneenterprise64.msi (NODE-1) [11-24-18 8:16:08 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome x64.transfer.NODE-1.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.17 [11-24-18 8:16:08 pm] | Started sync for Snapin Chrome x64 - Resource id #761 [11-24-18 8:16:08 pm] # Chrome x64: File does not exist googlechromebetastandaloneenterprise64.msi (NODE-2) [11-24-18 8:16:08 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome x64.transfer.NODE-2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.18 [11-24-18 8:16:08 pm] | Started sync for Snapin Chrome x64 - Resource id #770 [11-24-18 8:16:09 pm] | Sync finished - Resource id #761 [11-24-18 8:16:09 pm] | Sync finished - Resource id #770 [11-24-18 8:26:04 pm] * Starting Snapin Replication. [11-24-18 8:26:04 pm] * We are group ID: 1. We are group name: default [11-24-18 8:26:04 pm] * We are node ID: 1. We are node name: DefaultMember [11-24-18 8:26:04 pm] * Attempting to perform Group -> Group snapin replication. [11-24-18 8:26:04 pm] | Replicating ssl less private key [11-24-18 8:26:04 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:26:04 pm] | File Name: ssl/fog.csr [11-24-18 8:26:05 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-1) [11-24-18 8:26:05 pm] * All files synced for this item. [11-24-18 8:26:05 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-2) [11-24-18 8:26:05 pm] * All files synced for this item. [11-24-18 8:26:06 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:26:06 pm] | File Name: ssl/CA [11-24-18 8:26:06 pm] * All files synced for this item. [11-24-18 8:26:06 pm] * All files synced for this item. [11-24-18 8:26:07 pm] * Not syncing Snapin between groups [11-24-18 8:26:07 pm] | Snapin Name: Chrome x64 [11-24-18 8:26:07 pm] | There are no other members to sync to. [11-24-18 8:26:07 pm] * Attempting to perform Group -> Nodes snapin replication. [11-24-18 8:26:07 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:26:07 pm] | Snapin Name: Chrome x64 [11-24-18 8:26:08 pm] # Chrome x64: No need to sync googlechromebetastandaloneenterprise64.msi (NODE-1) [11-24-18 8:26:08 pm] * All files synced for this item. [11-24-18 8:26:09 pm] # Chrome x64: No need to sync googlechromebetastandaloneenterprise64.msi (NODE-2) [11-24-18 8:26:09 pm] * All files synced for this item. [11-24-18 8:36:04 pm] * Starting Snapin Replication. [11-24-18 8:36:04 pm] * We are group ID: 1. We are group name: default [11-24-18 8:36:04 pm] * We are node ID: 1. We are node name: DefaultMember [11-24-18 8:36:04 pm] * Attempting to perform Group -> Group snapin replication. [11-24-18 8:36:04 pm] | Replicating ssl less private key [11-24-18 8:36:04 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:36:04 pm] | File Name: ssl/fog.csr [11-24-18 8:36:05 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-1) [11-24-18 8:36:05 pm] * All files synced for this item. [11-24-18 8:36:06 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-2) [11-24-18 8:36:06 pm] * All files synced for this item. [11-24-18 8:36:06 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:36:06 pm] | File Name: ssl/CA [11-24-18 8:36:06 pm] * All files synced for this item. [11-24-18 8:36:07 pm] * All files synced for this item. [11-24-18 8:36:07 pm] * Not syncing Snapin between groups [11-24-18 8:36:07 pm] | Snapin Name: Chrome Beta [11-24-18 8:36:07 pm] | There are no other members to sync to. [11-24-18 8:36:07 pm] * Not syncing Snapin between groups [11-24-18 8:36:07 pm] | Snapin Name: Chrome x64 [11-24-18 8:36:07 pm] | There are no other members to sync to. [11-24-18 8:36:07 pm] * Attempting to perform Group -> Nodes snapin replication. [11-24-18 8:36:08 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:36:08 pm] | Snapin Name: Chrome Beta [11-24-18 8:36:08 pm] # Chrome Beta: File size mismatch - googlechromebetastandaloneenterprise64.msi: 0 != 56467664 [11-24-18 8:36:08 pm] # Chrome Beta: Deleting remote file googlechromebetastandaloneenterprise64.msi [11-24-18 8:36:08 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome Beta.transfer.NODE-1.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.17 [11-24-18 8:36:08 pm] | Started sync for Snapin Chrome Beta - Resource id #1058 [11-24-18 8:36:09 pm] # Chrome Beta: File size mismatch - googlechromebetastandaloneenterprise64.msi: 0 != 56467664 [11-24-18 8:36:09 pm] # Chrome Beta: Deleting remote file googlechromebetastandaloneenterprise64.msi [11-24-18 8:36:09 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome Beta.transfer.NODE-2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.18 [11-24-18 8:36:09 pm] | Started sync for Snapin Chrome Beta - Resource id #1071 [11-24-18 8:36:09 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:36:09 pm] | Snapin Name: Chrome x64 [11-24-18 8:36:10 pm] # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-1) [11-24-18 8:36:10 pm] * All files synced for this item. [11-24-18 8:36:11 pm] # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-2) [11-24-18 8:36:11 pm] * All files synced for this item. [11-24-18 8:36:11 pm] | Sync finished - Resource id #1058 [11-24-18 8:36:11 pm] | Sync finished - Resource id #1071 [11-24-18 8:46:04 pm] * Starting Snapin Replication. [11-24-18 8:46:04 pm] * We are group ID: 1. We are group name: default [11-24-18 8:46:04 pm] * We are node ID: 1. We are node name: DefaultMember [11-24-18 8:46:04 pm] * Attempting to perform Group -> Group snapin replication. [11-24-18 8:46:04 pm] | Replicating ssl less private key [11-24-18 8:46:04 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:46:04 pm] | File Name: ssl/fog.csr [11-24-18 8:46:05 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-1) [11-24-18 8:46:05 pm] * All files synced for this item. [11-24-18 8:46:05 pm] # ssl/fog.csr: No need to sync fog.csr (NODE-2) [11-24-18 8:46:05 pm] * All files synced for this item. [11-24-18 8:46:06 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:46:06 pm] | File Name: ssl/CA [11-24-18 8:46:06 pm] * All files synced for this item. [11-24-18 8:46:06 pm] * All files synced for this item. [11-24-18 8:46:07 pm] * Not syncing Snapin between groups [11-24-18 8:46:07 pm] | Snapin Name: Chrome Beta [11-24-18 8:46:07 pm] | There are no other members to sync to. [11-24-18 8:46:07 pm] * Not syncing Snapin between groups [11-24-18 8:46:07 pm] | Snapin Name: Chrome x64 [11-24-18 8:46:07 pm] | There are no other members to sync to. [11-24-18 8:46:07 pm] * Attempting to perform Group -> Nodes snapin replication. [11-24-18 8:46:07 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:46:07 pm] | Snapin Name: Chrome Beta [11-24-18 8:46:08 pm] # Chrome Beta: File size mismatch - googlechromebetastandaloneenterprise64.msi: 56467664 != 0 [11-24-18 8:46:08 pm] # Chrome Beta: Deleting remote file googlechromebetastandaloneenterprise64.msi [11-24-18 8:46:08 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome Beta.transfer.NODE-1.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.17 [11-24-18 8:46:08 pm] | Started sync for Snapin Chrome Beta - Resource id #1261 [11-24-18 8:46:08 pm] # Chrome Beta: File size mismatch - googlechromebetastandaloneenterprise64.msi: 56467664 != 0 [11-24-18 8:46:08 pm] # Chrome Beta: Deleting remote file googlechromebetastandaloneenterprise64.msi [11-24-18 8:46:08 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file /opt/fog/log/fogsnapinrep.Chrome Beta.transfer.NODE-2.log;set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "googlechromebetastandaloneenterprise64.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 192.168.1.18 [11-24-18 8:46:08 pm] | Started sync for Snapin Chrome Beta - Resource id #1274 [11-24-18 8:46:09 pm] * Found Snapin to transfer to 2 nodes [11-24-18 8:46:09 pm] | Snapin Name: Chrome x64 [11-24-18 8:46:10 pm] # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-1) [11-24-18 8:46:10 pm] * All files synced for this item. [11-24-18 8:46:10 pm] # Chrome x64: No need to sync GoogleChromeStandaloneEnterprise64.msi (NODE-2) [11-24-18 8:46:10 pm] * All files synced for this item. [11-24-18 8:46:10 pm] | Sync finished - Resource id #1261 [11-24-18 8:46:10 pm] | Sync finished - Resource id #1274- This last issue is for both images and snapins, but is more based on the something I noticed in the web UI, so I’m not sure which dev to report this to as a possible bug, or if it’s more of a feature request…but if you go to delete one or more snapins / images from the list of all snapins / all images, it will not remove the files associated with whatever you delete. In order to do so, you must enter each image / snapin individually, go to the delete tab, and check the box to delete files when deleting. It would be nice to have on the all images / snapins pages for mass changes or just trying to explain to my team that only one path of deleting gives expected results.
Please put this option:
Here:
Sorry if it seems like I am nitpicking, I am just trying to be thorough in my testing and ran tests based off of issues I have come across myself by accident. I am sure there are users who will never encounter the scenarios I have provided, but I am trying to cover any possible future headaches. Yay QA testing…
-
-
@jflippen Thanks heaps for testing and your lengthy report, highly appreciated! Now about your questions/findings:
-
Good point. So essentially the snapin binaries pile up over time. On the other hand I am wondering if we should simply remove any unknown files from the snapin directory? What if we delete a file which someone put there by intention and didn’t want us to delete it? The best way I find would be to ask the user if we should delete the old binary when editing a snapin takes place and a different filename is used. This way people can decide if they want to keep the old binary on the server (might use it later) or not. What do you think? I created a github issue to keep track of this.
-
As you said, this ties in very closely with 1. and I am tending to be conservative here and not just delete files we don’t know about. You are right that we sync and delete files within
/images/<IMAGENAME>/...sub directories. It’s too much to keep track of all individual files in there. But if you simply create a directory/images/test123/and put files into that, replication won’t touch it! Same with files in the snapin directory which are not associated anymore. -
Oh yeah, not good. Found and fixed (in
dev-branch). But seems like this bug has been there for over a year now. We definitely need more people to test FOG as well as developers to fix stuff.
-
Yeah would be more like a feature request. May I ask you to post this as a new topic in the forums (section feature requests) so it’s not lost here. But I won’t promise we’ll implement it. As already mentioned we are careful with deleting files via web interface. Especially this kind of “mass deletion”. Pleas post and we’ll discuss this with all the others.
-
-
-
Your idea sounds like a good compromise. Something like delete old file on new upload checkbox that is checked by default maybe?
-
okay, that makes sense. I forgot that the images folder only syncs folders tied to an image in the table.
-
I tried your update and that fixed the file getting set to 0kb. I test when I can, as FOG has a lot of potential in my eyes and I know most if not all of you are doing this in your spare time, which means a lot. I’m hoping someday I’ll have time to learn PHP and contribute more to the project, but that’s a battle for another day.
-
Posted in feature request and tagged you and this post for reference.
-