Increased network traffic since 1.5.0 upgrade (Snapins)



  • Server
    FOG Version: 1.5.0
    OS: Ubuntu 16.04

    Client
    Service Version: 0.11.15
    OS: Windows 10 Education 64bit

    I updated our FOG network from 1.4.4 to 1.5.0 last weekend (1 home server + 13 storage nodes) and have noticed a nearly steady 10mbps outgoing traffic from every storage node. It is all seems pointed at the main server and is creating a steady 100mbps +/- traffic stream on the incoming side of the server. When I watch tcpdump on the storage nodes i can see the normal mysql traffic but there is also a steady stream of outgoing FTP traffic to the main server. What I have done so far;

    -Checked permissions on the images and snapins directories.
    -Checked the image replication logs and all seems well and nothing stands out.
    -Checked the snapin replication logs and the only thing I see that jumps out at me is the continuous pattern like this:

    [03-06-18 9:33:22 pm]  * Found Snapin to transfer to 14 nodes
    [03-06-18 9:33:22 pm]  | Snapin Name: Cisco Jabber
    [03-06-18 9:33:22 pm]  | 46388736 0 /opt/fog/snapins/CiscoJabberSetup.msi ftp://fog:3AvpPHuzG5f0Mp2l2ma%2FBZjDFhvR6YqbsTNzZMNkHzo%3D@10.40.7.33/opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:33:22 pm]  | Files do not match.
    [03-06-18 9:33:22 pm]  * Deleting remote file: /opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:33:22 pm]  * Starting Sync Actions
    [03-06-18 9:33:22 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.40.7.33
    [03-06-18 9:33:22 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:27 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRECFOG01
    [03-06-18 9:33:27 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.28.25.165
    [03-06-18 9:33:27 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:33 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IREMEFOG01
    [03-06-18 9:33:33 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.24.25.165
    [03-06-18 9:33:33 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:34 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGMSFOG01
    [03-06-18 9:33:34 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.25.25.165
    [03-06-18 9:33:34 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:39 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGWCFOG01
    [03-06-18 9:33:39 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.26.25.165
    [03-06-18 9:33:39 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:41 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRHTEFOG01
    [03-06-18 9:33:41 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.31.25.165
    [03-06-18 9:33:41 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:46 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRJCEFOG01
    [03-06-18 9:33:46 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.29.25.165
    [03-06-18 9:33:46 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:51 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLBEFOG01
    [03-06-18 9:33:51 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.32.25.165
    [03-06-18 9:33:51 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:57 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLNEFOG01
    [03-06-18 9:33:57 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.22.25.165
    [03-06-18 9:33:57 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:02 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRMMSFOG01
    [03-06-18 9:34:02 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.21.25.165
    [03-06-18 9:34:02 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:03 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRNGEFOG01
    [03-06-18 9:34:03 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.46.25.165
    [03-06-18 9:34:03 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:08 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRPSEFOG01
    [03-06-18 9:34:08 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.30.25.165
    [03-06-18 9:34:08 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:10 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRSCHSFOG01
    [03-06-18 9:34:10 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.39.25.165
    [03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:10 pm]  | 46388736 0 /opt/fog/snapins/CiscoJabberSetup.msi ftp://fog:ejOVmFXgU4rbglxcETxqt7AVI%2BBh6h%2BDnWv0ksrwBpA%3D@10.27.25.165/opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:34:10 pm]  | Files do not match.
    [03-06-18 9:34:10 pm]  * Deleting remote file: /opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:34:10 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.27.25.165
    [03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber
    

    It repeats for every snapin on every run. There is a line stating “files don’t match”, this is what made me think there was a permissions change made during the upgrade(s) but that does not seem to be the case as it then states no need to sync. I have deleted all the snapins off a couple of nodes and let them replicate again and after the initial replication they start showing the same log messages again on the next run. I reloaded a couple of snapin files on the main server thinking possibly a file hash issue was created during the upgrade but same results.

    If I set all snapins to not replicate the traffic goes to almost nothing which is what I am accustomed to seeing on a normal basis prior to the upgrade.

    Edit: None are running in a VM environment, all are on bare metal.

    I shut off all snapin replication at 11:15 and the traffic just dropped as seen below. The IRHS - FOG is the main server and all the others are storage nodes.

    0_1520397204531_Firefox_Screenshot_2018-03-07T04-20-50.875Z.png



  • @Sebastian-Roth I have updated everything to the latest development version without issue tonight. The web gui definitely seems faster. I will watch for any oddities tomorrow while the system is in use. Thank You Guys again for all you do!


  • Developer

    @dclark I just merged the stuff into dev-branch an hour ago. Give it a try and see what you think of it.



  • @Sebastian-Roth Just let me know when you have it a point you are comfortable with trying it and I will certainly deploy the latest development branch.


  • Developer

    @dclark good you went to the dev-branch as I have fixed a snapin replication problem there which we had in the 1.5.5 release…

    As you have a fair amount of storage nodes you might want to test the updates I will push to the dev-branch fairly soon?! Will improve the web UI speed use even less traffic between the nodes.



  • @Sebastian-Roth Went through the day today without any reported issues and network activity has been at a minimum with the FOG communications between the server and nodes. Great Work Guys!


  • Developer

    @dclark said in Increased network traffic since 1.5.0 upgrade (Snapins):

    Looks like it was updated in 1.5.5

    Yeah, we fixed that! :-)



  • @Sebastian-Roth I have updated the entire network to the 1.5.5.1 Development Branch tonight. So far things seem to be stable and showing much less network traffic between the server and storage nodes than I was showing previously. There was a 10-15mb outgoing heartbeat from each node to the server when replication was enabled that does not seem to be present now.

    My only other concern would be the update to the kernel to solve the partition deletion issue during imaging that was taking so long, I had reverted back to the earlier version when using 1.5.0. Looks like it was updated in 1.5.5 but I will not be able to verify until tomorrow when I get into work.


  • Developer

    @dclark Just found this by accident as I have been working on the replication code a fair bit. You might want to update to the latest dev-branch. I have fixed all known replication issues in the last weeks.

    Important:
    All nodes need to be updated at the same time or images will be replicated over and over again as some of the hashing code needed to be changed. Please make sure to stop replication on the master first (systemctl stop FOGImageReplicator; systemctl stop FOGSnapinReplicator), then update the storage node(s) and then update master node as a last step.



  • It may or may not be related but I ran into the dashboard and storage pages taking minutes to load yesterday. We could get to the login screen and start the login process but the dashboard was taking up to 4 minutes to load. We have seen a slight slowness in these screens since updating to 1.5.0 but not like this. We have 13 storage nodes and one happened to be down yesterday which I believe that the disk usage graph on the dashboard was waiting for a response from the node that was down. I disabled all the graphs for everything except the main server and the dashboard loads just as quick as the other pages now. The storage page was still loading very slow so I deleted the storage node that was down and that page again loads, slower than the rest but in a reasonable amount of time.

    Is it possible that there is a timeout check somewhere for storage nodes that could be set lower, and is it possible that the traffic issue is related in some way waiting for a response that never comes?



  • @Tom-Elliott I have installed the working branch (shows up as 1.5.0.7) on the main server (IRHSFOG01) and two storage nodes (IRSMSFOG01 and IRSCHSFOG01) and have not seen any change from the stable version when I set the snapins to replicate. I also have one storage node (10.40.7.33) as a test running 1.4.4 and it displays the same behavior.

    I have attached the start of the new log file for the snapin replicator through the first snapin.

    Let me know if there are any other logs or packet traces you would like or if you have any ideas on other ways I can help to track this down.

    [03-07-18 9:20:12 pm] 
    ==================================
    ===        ====    =====      ====
    ===  =========  ==  ===   ==   ===
    ===  ========  ====  ==  ====  ===
    ===  ========  ====  ==  =========
    ===      ====  ====  ==  =========
    ===  ========  ====  ==  ===   ===
    ===  ========  ====  ==  ====  ===
    ===  =========  ==  ===   ==   ===
    ===  ==========    =====      ====
    ==================================
    ===== Free Opensource Ghost ======
    ==================================
    ============ Credits =============
    = https://fogproject.org/Credits =
    ==================================
    == Released under GPL Version 3 ==
    ==================================
    
    
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: 10.40.8.183
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: 10.40.25.165
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: 127.0.0.1
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: 127.0.1.1
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: xxxxxxxxxxxx
    [03-07-18 9:20:12 pm]  * Starting SnapinReplicator Service
    [03-07-18 9:20:12 pm]  * Checking for new items every 600 seconds
    [03-07-18 9:20:12 pm]  * Starting service loop
    [03-07-18 9:20:12 pm]  * Starting Snapin Replication.
    [03-07-18 9:20:12 pm]  * We are group ID: 1. We are group name: default
    [03-07-18 9:20:12 pm]  * We are node ID: 21. We are node name: IRHSFOG01
    [03-07-18 9:20:12 pm]  * Attempting to perform Group -> Group snapin replication.
    [03-07-18 9:20:12 pm]  | There is nothing to replicate.
    [03-07-18 9:20:12 pm]  | Please physically associate snapins to a storage group.
    [03-07-18 9:30:12 pm]  * Starting Snapin Replication.
    [03-07-18 9:30:12 pm]  * We are group ID: 1. We are group name: default
    [03-07-18 9:30:12 pm]  * We are node ID: 21. We are node name: IRHSFOG01
    [03-07-18 9:30:12 pm]  * Attempting to perform Group -> Group snapin replication.
    [03-07-18 9:30:12 pm]  | Replicating ssl less private key
    [03-07-18 9:30:12 pm]  * Found Snapin to transfer to 14 nodes
    [03-07-18 9:30:12 pm]  | File Name: ssl/fog.csr
    [03-07-18 9:30:12 pm]  | ssl/fog.csr: No need to sync fog.csr file to 10.40.7.33
    [03-07-18 9:30:12 pm]  * Starting Sync Actions
    [03-07-18 9:30:12 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.40.7.33
    [03-07-18 9:30:12 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:13 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRECFOG01
    [03-07-18 9:30:13 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.28.25.165
    [03-07-18 9:30:13 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:14 pm]  | ssl/fog.csr: No need to sync fog.csr file to IREMEFOG01
    [03-07-18 9:30:14 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.24.25.165
    [03-07-18 9:30:14 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:16 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRGMSFOG01
    [03-07-18 9:30:16 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.25.25.165
    [03-07-18 9:30:16 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:17 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRGWCFOG01
    [03-07-18 9:30:17 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.26.25.165
    [03-07-18 9:30:17 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:17 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRHTEFOG01
    [03-07-18 9:30:17 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.31.25.165
    [03-07-18 9:30:17 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:18 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRJCEFOG01
    [03-07-18 9:30:18 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.29.25.165
    [03-07-18 9:30:18 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:19 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRLBEFOG01
    [03-07-18 9:30:19 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.32.25.165
    [03-07-18 9:30:19 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:20 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRLNEFOG01
    [03-07-18 9:30:20 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.22.25.165
    [03-07-18 9:30:20 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:20 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRMMSFOG01
    [03-07-18 9:30:20 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.21.25.165
    [03-07-18 9:30:20 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:21 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRNGEFOG01
    [03-07-18 9:30:21 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.46.25.165
    [03-07-18 9:30:21 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:22 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRPSEFOG01
    [03-07-18 9:30:22 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.30.25.165
    [03-07-18 9:30:22 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:23 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRSCHSFOG01
    [03-07-18 9:30:23 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.39.25.165
    [03-07-18 9:30:23 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:23 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRSMSFOG01
    [03-07-18 9:30:23 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.27.25.165
    [03-07-18 9:30:23 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:23 pm]  * Found Snapin to transfer to 14 nodes
    [03-07-18 9:30:23 pm]  | File Name: ssl/CA
    [03-07-18 9:30:24 pm]  * Starting Sync Actions
    [03-07-18 9:30:24 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.40.7.33
    [03-07-18 9:30:24 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:24 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.28.25.165
    [03-07-18 9:30:24 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:24 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.24.25.165
    [03-07-18 9:30:24 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:25 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.25.25.165
    [03-07-18 9:30:25 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:25 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.26.25.165
    [03-07-18 9:30:25 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:25 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.31.25.165
    [03-07-18 9:30:25 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:26 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.29.25.165
    [03-07-18 9:30:26 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:26 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.32.25.165
    [03-07-18 9:30:26 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:26 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.22.25.165
    [03-07-18 9:30:26 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:27 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.21.25.165
    [03-07-18 9:30:27 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:27 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.46.25.165
    [03-07-18 9:30:27 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:27 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.30.25.165
    [03-07-18 9:30:27 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:28 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.39.25.165
    [03-07-18 9:30:28 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:28 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.27.25.165
    [03-07-18 9:30:28 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Cisco Jabber
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Dell 3020
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Dell GetBootOrder
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: DESAA Secure Browser 10.0
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: DisableRecovery
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Java 8-151
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Ladibug
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: NAF Secure Browser 2.1.1.0
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Office16
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: SEP 14
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Sketch up Pro 2017
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Software-Hardware Inventory
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - EME
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - GMS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - GWC
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - IRHS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - JCE
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - MMS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - NGE
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - PSE
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - SCHS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - SDSA
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - SMS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA -LNE
    [03-07-18 9:30:29 pm]  | There are no other members to sync to.
    [03-07-18 9:30:29 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:29 pm]  | Snapin Name: WIDA- LBE
    [03-07-18 9:30:29 pm]  | There are no other members to sync to.
    [03-07-18 9:30:29 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:29 pm]  | Snapin Name: Z - Win10-GuestLogin
    [03-07-18 9:30:29 pm]  | There are no other members to sync to.
    [03-07-18 9:30:29 pm]  * Attempting to perform Group -> Nodes snapin replication.
    [03-07-18 9:30:29 pm]  * Found Snapin to transfer to 14 nodes
    [03-07-18 9:30:29 pm]  | Snapin Name: Cisco Jabber
    [03-07-18 9:30:29 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to 10.40.7.33
    [03-07-18 9:30:29 pm]  * Starting Sync Actions
    [03-07-18 9:30:29 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.40.7.33
    [03-07-18 9:30:29 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:35 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRECFOG01
    [03-07-18 9:30:35 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.28.25.165
    [03-07-18 9:30:35 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:40 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IREMEFOG01
    [03-07-18 9:30:40 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.24.25.165
    [03-07-18 9:30:40 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:42 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGMSFOG01
    [03-07-18 9:30:42 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.25.25.165
    [03-07-18 9:30:42 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:47 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGWCFOG01
    [03-07-18 9:30:47 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.26.25.165
    [03-07-18 9:30:47 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:48 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRHTEFOG01
    [03-07-18 9:30:48 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.31.25.165
    [03-07-18 9:30:48 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:53 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRJCEFOG01
    [03-07-18 9:30:53 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.29.25.165
    [03-07-18 9:30:53 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:59 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLBEFOG01
    [03-07-18 9:30:59 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.32.25.165
    [03-07-18 9:30:59 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:04 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLNEFOG01
    [03-07-18 9:31:04 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.22.25.165
    [03-07-18 9:31:04 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:09 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRMMSFOG01
    [03-07-18 9:31:09 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.21.25.165
    [03-07-18 9:31:09 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:11 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRNGEFOG01
    [03-07-18 9:31:11 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.46.25.165
    [03-07-18 9:31:11 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:16 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRPSEFOG01
    [03-07-18 9:31:16 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.30.25.165
    [03-07-18 9:31:16 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:18 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRSCHSFOG01
    [03-07-18 9:31:18 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.39.25.165
    [03-07-18 9:31:18 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:23 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRSMSFOG01
    [03-07-18 9:31:23 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.27.25.165
    [03-07-18 9:31:23 pm]  * Started sync for Snapin Cisco Jabber```


  • I figured I had a little time prior to anyone else getting in and went ahead and updated the main server to the 1.5 working branch and enabled snapin replication again and am seeing the same behavior on all the nodes including the 1.4.4 test node. I did not update any nodes yet but thought I would give you the preliminary findings. I am going to roll back to the 1.5 release and will try updating both the main again and the nodes late today.

    You mention the web GUI being slow. The only page I notice any real slowness is the login page and the home page. Wondering if that isn’t an issue with nodes as well, I believe that it reaches out to each storage node on every main page load to get the disk space for the charts, I think?



  • Thanks for the quick response!

    I will try the 1.5.1 branch this afternoon after the main rush of the day is over just in case it gives us some undesired results I will be able to roll back without any real user impact.

    Just as a side note - I spun up a test storage node with the 1.4.4 version to see if it displayed the same behavior and it does when it is working with the production server running 1.5.0.


  • Senior Developer

    @dclark said in Increased network traffic since 1.5.0 upgrade (Snapins):

    [03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:10 pm]  | 46388736 0 /opt/fog/snapins/CiscoJabberSetup.msi ftp://fog:ejOVmFXgU4rbglxcETxqt7AVI%2BBh6h%2BDnWv0ksrwBpA%3D@10.27.25.165/opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:34:10 pm]  | Files do not match.
    [03-06-18 9:34:10 pm]  * Deleting remote file: /opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:34:10 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.27.25.165
    [03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber
    

    The above, to me, seems to indicate proper action. as you can see, the file sizes don’t match up, meaning either the remote node this was testing could not be contacted, or the file did not exist. (I say these because the filesize was 0 on the remote side.)

    [03-06-18 9:33:51 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:57 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLNEFOG01
    [03-06-18 9:33:57 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.22.25.165
    

    This line interests me as I don’t think i’ve ever seen this issue ever before. So while I’ve never seen it, I hope I may have found the reason to why it was happening and have a hopeful fix for it. Essentially my code was setting a test variable if it wasn’t already set. The idea was if other nodes needed the file, it would be able to send it. I guess this was an oversite? I’m not sure what impacts this will have, but I think it will work for the problem you were seeing.

    Please try installing the working-1.5.1 branch? I know it’s not in RC or anything, I’m only pushing bug fixes there. I don’t have multiple nodes to test if this is working so I’m just hoping your input/help would be okay as, like I said, you’re also the first i’ve seen of this particular issue.


  • Senior Developer

    @dclark thank you for gathering all this information!

    Cross linking with this thread: https://forums.fogproject.org/topic/11572/web-gui-slow-after-adding-storage-node as it sounds similar.


    @Tom-Elliott a replication bug would explain the high server traffic and the other user’s slow GUI


 

554
Online

5.4k
Users

12.6k
Topics

118.8k
Posts