Increased network traffic since 1.5.0 upgrade (Snapins)



  • Server
    FOG Version: 1.5.0
    OS: Ubuntu 16.04

    Client
    Service Version: 0.11.15
    OS: Windows 10 Education 64bit

    I updated our FOG network from 1.4.4 to 1.5.0 last weekend (1 home server + 13 storage nodes) and have noticed a nearly steady 10mbps outgoing traffic from every storage node. It is all seems pointed at the main server and is creating a steady 100mbps +/- traffic stream on the incoming side of the server. When I watch tcpdump on the storage nodes i can see the normal mysql traffic but there is also a steady stream of outgoing FTP traffic to the main server. What I have done so far;

    -Checked permissions on the images and snapins directories.
    -Checked the image replication logs and all seems well and nothing stands out.
    -Checked the snapin replication logs and the only thing I see that jumps out at me is the continuous pattern like this:

    [03-06-18 9:33:22 pm]  * Found Snapin to transfer to 14 nodes
    [03-06-18 9:33:22 pm]  | Snapin Name: Cisco Jabber
    [03-06-18 9:33:22 pm]  | 46388736 0 /opt/fog/snapins/CiscoJabberSetup.msi ftp://fog:3AvpPHuzG5f0Mp2l2ma%2FBZjDFhvR6YqbsTNzZMNkHzo%3D@10.40.7.33/opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:33:22 pm]  | Files do not match.
    [03-06-18 9:33:22 pm]  * Deleting remote file: /opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:33:22 pm]  * Starting Sync Actions
    [03-06-18 9:33:22 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.40.7.33
    [03-06-18 9:33:22 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:27 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRECFOG01
    [03-06-18 9:33:27 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.28.25.165
    [03-06-18 9:33:27 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:33 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IREMEFOG01
    [03-06-18 9:33:33 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.24.25.165
    [03-06-18 9:33:33 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:34 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGMSFOG01
    [03-06-18 9:33:34 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.25.25.165
    [03-06-18 9:33:34 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:39 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGWCFOG01
    [03-06-18 9:33:39 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.26.25.165
    [03-06-18 9:33:39 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:41 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRHTEFOG01
    [03-06-18 9:33:41 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.31.25.165
    [03-06-18 9:33:41 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:46 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRJCEFOG01
    [03-06-18 9:33:46 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.29.25.165
    [03-06-18 9:33:46 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:51 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLBEFOG01
    [03-06-18 9:33:51 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.32.25.165
    [03-06-18 9:33:51 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:57 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLNEFOG01
    [03-06-18 9:33:57 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.22.25.165
    [03-06-18 9:33:57 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:02 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRMMSFOG01
    [03-06-18 9:34:02 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.21.25.165
    [03-06-18 9:34:02 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:03 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRNGEFOG01
    [03-06-18 9:34:03 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.46.25.165
    [03-06-18 9:34:03 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:08 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRPSEFOG01
    [03-06-18 9:34:08 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.30.25.165
    [03-06-18 9:34:08 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:10 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRSCHSFOG01
    [03-06-18 9:34:10 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.39.25.165
    [03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:10 pm]  | 46388736 0 /opt/fog/snapins/CiscoJabberSetup.msi ftp://fog:ejOVmFXgU4rbglxcETxqt7AVI%2BBh6h%2BDnWv0ksrwBpA%3D@10.27.25.165/opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:34:10 pm]  | Files do not match.
    [03-06-18 9:34:10 pm]  * Deleting remote file: /opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:34:10 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.27.25.165
    [03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber
    

    It repeats for every snapin on every run. There is a line stating “files don’t match”, this is what made me think there was a permissions change made during the upgrade(s) but that does not seem to be the case as it then states no need to sync. I have deleted all the snapins off a couple of nodes and let them replicate again and after the initial replication they start showing the same log messages again on the next run. I reloaded a couple of snapin files on the main server thinking possibly a file hash issue was created during the upgrade but same results.

    If I set all snapins to not replicate the traffic goes to almost nothing which is what I am accustomed to seeing on a normal basis prior to the upgrade.

    Edit: None are running in a VM environment, all are on bare metal.

    I shut off all snapin replication at 11:15 and the traffic just dropped as seen below. The IRHS - FOG is the main server and all the others are storage nodes.

    0_1520397204531_Firefox_Screenshot_2018-03-07T04-20-50.875Z.png



  • It may or may not be related but I ran into the dashboard and storage pages taking minutes to load yesterday. We could get to the login screen and start the login process but the dashboard was taking up to 4 minutes to load. We have seen a slight slowness in these screens since updating to 1.5.0 but not like this. We have 13 storage nodes and one happened to be down yesterday which I believe that the disk usage graph on the dashboard was waiting for a response from the node that was down. I disabled all the graphs for everything except the main server and the dashboard loads just as quick as the other pages now. The storage page was still loading very slow so I deleted the storage node that was down and that page again loads, slower than the rest but in a reasonable amount of time.

    Is it possible that there is a timeout check somewhere for storage nodes that could be set lower, and is it possible that the traffic issue is related in some way waiting for a response that never comes?



  • @Tom-Elliott I have installed the working branch (shows up as 1.5.0.7) on the main server (IRHSFOG01) and two storage nodes (IRSMSFOG01 and IRSCHSFOG01) and have not seen any change from the stable version when I set the snapins to replicate. I also have one storage node (10.40.7.33) as a test running 1.4.4 and it displays the same behavior.

    I have attached the start of the new log file for the snapin replicator through the first snapin.

    Let me know if there are any other logs or packet traces you would like or if you have any ideas on other ways I can help to track this down.

    [03-07-18 9:20:12 pm] 
    ==================================
    ===        ====    =====      ====
    ===  =========  ==  ===   ==   ===
    ===  ========  ====  ==  ====  ===
    ===  ========  ====  ==  =========
    ===      ====  ====  ==  =========
    ===  ========  ====  ==  ===   ===
    ===  ========  ====  ==  ====  ===
    ===  =========  ==  ===   ==   ===
    ===  ==========    =====      ====
    ==================================
    ===== Free Opensource Ghost ======
    ==================================
    ============ Credits =============
    = https://fogproject.org/Credits =
    ==================================
    == Released under GPL Version 3 ==
    ==================================
    
    
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: 10.40.8.183
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: 10.40.25.165
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: 127.0.0.1
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: 127.0.1.1
    [03-07-18 9:20:12 pm] Interface Ready with IP Address: xxxxxxxxxxxx
    [03-07-18 9:20:12 pm]  * Starting SnapinReplicator Service
    [03-07-18 9:20:12 pm]  * Checking for new items every 600 seconds
    [03-07-18 9:20:12 pm]  * Starting service loop
    [03-07-18 9:20:12 pm]  * Starting Snapin Replication.
    [03-07-18 9:20:12 pm]  * We are group ID: 1. We are group name: default
    [03-07-18 9:20:12 pm]  * We are node ID: 21. We are node name: IRHSFOG01
    [03-07-18 9:20:12 pm]  * Attempting to perform Group -> Group snapin replication.
    [03-07-18 9:20:12 pm]  | There is nothing to replicate.
    [03-07-18 9:20:12 pm]  | Please physically associate snapins to a storage group.
    [03-07-18 9:30:12 pm]  * Starting Snapin Replication.
    [03-07-18 9:30:12 pm]  * We are group ID: 1. We are group name: default
    [03-07-18 9:30:12 pm]  * We are node ID: 21. We are node name: IRHSFOG01
    [03-07-18 9:30:12 pm]  * Attempting to perform Group -> Group snapin replication.
    [03-07-18 9:30:12 pm]  | Replicating ssl less private key
    [03-07-18 9:30:12 pm]  * Found Snapin to transfer to 14 nodes
    [03-07-18 9:30:12 pm]  | File Name: ssl/fog.csr
    [03-07-18 9:30:12 pm]  | ssl/fog.csr: No need to sync fog.csr file to 10.40.7.33
    [03-07-18 9:30:12 pm]  * Starting Sync Actions
    [03-07-18 9:30:12 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.40.7.33
    [03-07-18 9:30:12 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:13 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRECFOG01
    [03-07-18 9:30:13 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.28.25.165
    [03-07-18 9:30:13 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:14 pm]  | ssl/fog.csr: No need to sync fog.csr file to IREMEFOG01
    [03-07-18 9:30:14 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.24.25.165
    [03-07-18 9:30:14 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:16 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRGMSFOG01
    [03-07-18 9:30:16 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.25.25.165
    [03-07-18 9:30:16 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:17 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRGWCFOG01
    [03-07-18 9:30:17 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.26.25.165
    [03-07-18 9:30:17 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:17 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRHTEFOG01
    [03-07-18 9:30:17 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.31.25.165
    [03-07-18 9:30:17 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:18 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRJCEFOG01
    [03-07-18 9:30:18 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.29.25.165
    [03-07-18 9:30:18 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:19 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRLBEFOG01
    [03-07-18 9:30:19 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.32.25.165
    [03-07-18 9:30:19 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:20 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRLNEFOG01
    [03-07-18 9:30:20 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.22.25.165
    [03-07-18 9:30:20 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:20 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRMMSFOG01
    [03-07-18 9:30:20 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.21.25.165
    [03-07-18 9:30:20 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:21 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRNGEFOG01
    [03-07-18 9:30:21 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.46.25.165
    [03-07-18 9:30:21 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:22 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRPSEFOG01
    [03-07-18 9:30:22 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.30.25.165
    [03-07-18 9:30:22 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:23 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRSCHSFOG01
    [03-07-18 9:30:23 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.39.25.165
    [03-07-18 9:30:23 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:23 pm]  | ssl/fog.csr: No need to sync fog.csr file to IRSMSFOG01
    [03-07-18 9:30:23 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.27.25.165
    [03-07-18 9:30:23 pm]  * Started sync for Snapin ssl/fog.csr
    [03-07-18 9:30:23 pm]  * Found Snapin to transfer to 14 nodes
    [03-07-18 9:30:23 pm]  | File Name: ssl/CA
    [03-07-18 9:30:24 pm]  * Starting Sync Actions
    [03-07-18 9:30:24 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.40.7.33
    [03-07-18 9:30:24 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:24 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.28.25.165
    [03-07-18 9:30:24 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:24 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.24.25.165
    [03-07-18 9:30:24 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:25 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.25.25.165
    [03-07-18 9:30:25 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:25 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.26.25.165
    [03-07-18 9:30:25 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:25 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.31.25.165
    [03-07-18 9:30:25 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:26 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.29.25.165
    [03-07-18 9:30:26 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:26 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.32.25.165
    [03-07-18 9:30:26 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:26 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.22.25.165
    [03-07-18 9:30:26 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:27 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.21.25.165
    [03-07-18 9:30:27 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:27 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.46.25.165
    [03-07-18 9:30:27 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:27 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.30.25.165
    [03-07-18 9:30:27 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:28 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.39.25.165
    [03-07-18 9:30:28 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:28 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.27.25.165
    [03-07-18 9:30:28 pm]  * Started sync for Snapin ssl/CA
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Cisco Jabber
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Dell 3020
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Dell GetBootOrder
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: DESAA Secure Browser 10.0
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: DisableRecovery
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Java 8-151
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Ladibug
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: NAF Secure Browser 2.1.1.0
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Office16
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: SEP 14
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Sketch up Pro 2017
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: Software-Hardware Inventory
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - EME
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - GMS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - GWC
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - IRHS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - JCE
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - MMS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - NGE
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - PSE
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - SCHS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - SDSA
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA - SMS
    [03-07-18 9:30:28 pm]  | There are no other members to sync to.
    [03-07-18 9:30:28 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:28 pm]  | Snapin Name: WIDA -LNE
    [03-07-18 9:30:29 pm]  | There are no other members to sync to.
    [03-07-18 9:30:29 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:29 pm]  | Snapin Name: WIDA- LBE
    [03-07-18 9:30:29 pm]  | There are no other members to sync to.
    [03-07-18 9:30:29 pm]  * Not syncing Snapin between groups
    [03-07-18 9:30:29 pm]  | Snapin Name: Z - Win10-GuestLogin
    [03-07-18 9:30:29 pm]  | There are no other members to sync to.
    [03-07-18 9:30:29 pm]  * Attempting to perform Group -> Nodes snapin replication.
    [03-07-18 9:30:29 pm]  * Found Snapin to transfer to 14 nodes
    [03-07-18 9:30:29 pm]  | Snapin Name: Cisco Jabber
    [03-07-18 9:30:29 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to 10.40.7.33
    [03-07-18 9:30:29 pm]  * Starting Sync Actions
    [03-07-18 9:30:29 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.40.7.33
    [03-07-18 9:30:29 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:35 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRECFOG01
    [03-07-18 9:30:35 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.28.25.165
    [03-07-18 9:30:35 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:40 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IREMEFOG01
    [03-07-18 9:30:40 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.24.25.165
    [03-07-18 9:30:40 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:42 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGMSFOG01
    [03-07-18 9:30:42 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.25.25.165
    [03-07-18 9:30:42 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:47 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGWCFOG01
    [03-07-18 9:30:47 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.26.25.165
    [03-07-18 9:30:47 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:48 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRHTEFOG01
    [03-07-18 9:30:48 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.31.25.165
    [03-07-18 9:30:48 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:53 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRJCEFOG01
    [03-07-18 9:30:53 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.29.25.165
    [03-07-18 9:30:53 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:30:59 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLBEFOG01
    [03-07-18 9:30:59 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.32.25.165
    [03-07-18 9:30:59 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:04 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLNEFOG01
    [03-07-18 9:31:04 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.22.25.165
    [03-07-18 9:31:04 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:09 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRMMSFOG01
    [03-07-18 9:31:09 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.21.25.165
    [03-07-18 9:31:09 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:11 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRNGEFOG01
    [03-07-18 9:31:11 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.46.25.165
    [03-07-18 9:31:11 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:16 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRPSEFOG01
    [03-07-18 9:31:16 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.30.25.165
    [03-07-18 9:31:16 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:18 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRSCHSFOG01
    [03-07-18 9:31:18 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.39.25.165
    [03-07-18 9:31:18 pm]  * Started sync for Snapin Cisco Jabber
    [03-07-18 9:31:23 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRSMSFOG01
    [03-07-18 9:31:23 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.27.25.165
    [03-07-18 9:31:23 pm]  * Started sync for Snapin Cisco Jabber```


  • I figured I had a little time prior to anyone else getting in and went ahead and updated the main server to the 1.5 working branch and enabled snapin replication again and am seeing the same behavior on all the nodes including the 1.4.4 test node. I did not update any nodes yet but thought I would give you the preliminary findings. I am going to roll back to the 1.5 release and will try updating both the main again and the nodes late today.

    You mention the web GUI being slow. The only page I notice any real slowness is the login page and the home page. Wondering if that isn’t an issue with nodes as well, I believe that it reaches out to each storage node on every main page load to get the disk space for the charts, I think?



  • Thanks for the quick response!

    I will try the 1.5.1 branch this afternoon after the main rush of the day is over just in case it gives us some undesired results I will be able to roll back without any real user impact.

    Just as a side note - I spun up a test storage node with the 1.4.4 version to see if it displayed the same behavior and it does when it is working with the production server running 1.5.0.


  • Senior Developer

    @dclark said in Increased network traffic since 1.5.0 upgrade (Snapins):

    [03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:34:10 pm]  | 46388736 0 /opt/fog/snapins/CiscoJabberSetup.msi ftp://fog:ejOVmFXgU4rbglxcETxqt7AVI%2BBh6h%2BDnWv0ksrwBpA%3D@10.27.25.165/opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:34:10 pm]  | Files do not match.
    [03-06-18 9:34:10 pm]  * Deleting remote file: /opt/fog/snapins/CiscoJabberSetup.msi
    [03-06-18 9:34:10 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.27.25.165
    [03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber
    

    The above, to me, seems to indicate proper action. as you can see, the file sizes don’t match up, meaning either the remote node this was testing could not be contacted, or the file did not exist. (I say these because the filesize was 0 on the remote side.)

    [03-06-18 9:33:51 pm]  * Started sync for Snapin Cisco Jabber
    [03-06-18 9:33:57 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLNEFOG01
    [03-06-18 9:33:57 pm]  | CMD:
    			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.22.25.165
    

    This line interests me as I don’t think i’ve ever seen this issue ever before. So while I’ve never seen it, I hope I may have found the reason to why it was happening and have a hopeful fix for it. Essentially my code was setting a test variable if it wasn’t already set. The idea was if other nodes needed the file, it would be able to send it. I guess this was an oversite? I’m not sure what impacts this will have, but I think it will work for the problem you were seeing.

    Please try installing the working-1.5.1 branch? I know it’s not in RC or anything, I’m only pushing bug fixes there. I don’t have multiple nodes to test if this is working so I’m just hoping your input/help would be okay as, like I said, you’re also the first i’ve seen of this particular issue.


  • Senior Developer

    @dclark thank you for gathering all this information!

    Cross linking with this thread: https://forums.fogproject.org/topic/11572/web-gui-slow-after-adding-storage-node as it sounds similar.


    @Tom-Elliott a replication bug would explain the high server traffic and the other user’s slow GUI


 

356
Online

41.8k
Users

12.3k
Topics

116.0k
Posts