Increased network traffic since 1.5.0 upgrade (Snapins)
-
@dclark thank you for gathering all this information!
Cross linking with this thread: https://forums.fogproject.org/topic/11572/web-gui-slow-after-adding-storage-node as it sounds similar.
@Tom-Elliott a replication bug would explain the high server traffic and the other user’s slow GUI
-
@dclark said in Increased network traffic since 1.5.0 upgrade (Snapins):
[03-06-18 9:34:10 pm] * Started sync for Snapin Cisco Jabber [03-06-18 9:34:10 pm] | 46388736 0 /opt/fog/snapins/CiscoJabberSetup.msi ftp://fog:ejOVmFXgU4rbglxcETxqt7AVI%2BBh6h%2BDnWv0ksrwBpA%3D@10.27.25.165/opt/fog/snapins/CiscoJabberSetup.msi [03-06-18 9:34:10 pm] | Files do not match. [03-06-18 9:34:10 pm] * Deleting remote file: /opt/fog/snapins/CiscoJabberSetup.msi [03-06-18 9:34:10 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.27.25.165 [03-06-18 9:34:10 pm] * Started sync for Snapin Cisco JabberThe above, to me, seems to indicate proper action. as you can see, the file sizes don’t match up, meaning either the remote node this was testing could not be contacted, or the file did not exist. (I say these because the filesize was 0 on the remote side.)
[03-06-18 9:33:51 pm] * Started sync for Snapin Cisco Jabber [03-06-18 9:33:57 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLNEFOG01 [03-06-18 9:33:57 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.22.25.165This line interests me as I don’t think i’ve ever seen this issue ever before. So while I’ve never seen it, I hope I may have found the reason to why it was happening and have a hopeful fix for it. Essentially my code was setting a test variable if it wasn’t already set. The idea was if other nodes needed the file, it would be able to send it. I guess this was an oversite? I’m not sure what impacts this will have, but I think it will work for the problem you were seeing.
Please try installing the working-1.5.1 branch? I know it’s not in RC or anything, I’m only pushing bug fixes there. I don’t have multiple nodes to test if this is working so I’m just hoping your input/help would be okay as, like I said, you’re also the first i’ve seen of this particular issue.
-
Thanks for the quick response!
I will try the 1.5.1 branch this afternoon after the main rush of the day is over just in case it gives us some undesired results I will be able to roll back without any real user impact.
Just as a side note - I spun up a test storage node with the 1.4.4 version to see if it displayed the same behavior and it does when it is working with the production server running 1.5.0.
-
I figured I had a little time prior to anyone else getting in and went ahead and updated the main server to the 1.5 working branch and enabled snapin replication again and am seeing the same behavior on all the nodes including the 1.4.4 test node. I did not update any nodes yet but thought I would give you the preliminary findings. I am going to roll back to the 1.5 release and will try updating both the main again and the nodes late today.
You mention the web GUI being slow. The only page I notice any real slowness is the login page and the home page. Wondering if that isn’t an issue with nodes as well, I believe that it reaches out to each storage node on every main page load to get the disk space for the charts, I think?
-
@Tom-Elliott I have installed the working branch (shows up as 1.5.0.7) on the main server (IRHSFOG01) and two storage nodes (IRSMSFOG01 and IRSCHSFOG01) and have not seen any change from the stable version when I set the snapins to replicate. I also have one storage node (10.40.7.33) as a test running 1.4.4 and it displays the same behavior.
I have attached the start of the new log file for the snapin replicator through the first snapin.
Let me know if there are any other logs or packet traces you would like or if you have any ideas on other ways I can help to track this down.
[03-07-18 9:20:12 pm] ================================== === ==== ===== ==== === ========= == === == === === ======== ==== == ==== === === ======== ==== == ========= === ==== ==== == ========= === ======== ==== == === === === ======== ==== == ==== === === ========= == === == === === ========== ===== ==== ================================== ===== Free Opensource Ghost ====== ================================== ============ Credits ============= = https://fogproject.org/Credits = ================================== == Released under GPL Version 3 == ================================== [03-07-18 9:20:12 pm] Interface Ready with IP Address: 10.40.8.183 [03-07-18 9:20:12 pm] Interface Ready with IP Address: 10.40.25.165 [03-07-18 9:20:12 pm] Interface Ready with IP Address: 127.0.0.1 [03-07-18 9:20:12 pm] Interface Ready with IP Address: 127.0.1.1 [03-07-18 9:20:12 pm] Interface Ready with IP Address: xxxxxxxxxxxx [03-07-18 9:20:12 pm] * Starting SnapinReplicator Service [03-07-18 9:20:12 pm] * Checking for new items every 600 seconds [03-07-18 9:20:12 pm] * Starting service loop [03-07-18 9:20:12 pm] * Starting Snapin Replication. [03-07-18 9:20:12 pm] * We are group ID: 1. We are group name: default [03-07-18 9:20:12 pm] * We are node ID: 21. We are node name: IRHSFOG01 [03-07-18 9:20:12 pm] * Attempting to perform Group -> Group snapin replication. [03-07-18 9:20:12 pm] | There is nothing to replicate. [03-07-18 9:20:12 pm] | Please physically associate snapins to a storage group. [03-07-18 9:30:12 pm] * Starting Snapin Replication. [03-07-18 9:30:12 pm] * We are group ID: 1. We are group name: default [03-07-18 9:30:12 pm] * We are node ID: 21. We are node name: IRHSFOG01 [03-07-18 9:30:12 pm] * Attempting to perform Group -> Group snapin replication. [03-07-18 9:30:12 pm] | Replicating ssl less private key [03-07-18 9:30:12 pm] * Found Snapin to transfer to 14 nodes [03-07-18 9:30:12 pm] | File Name: ssl/fog.csr [03-07-18 9:30:12 pm] | ssl/fog.csr: No need to sync fog.csr file to 10.40.7.33 [03-07-18 9:30:12 pm] * Starting Sync Actions [03-07-18 9:30:12 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.40.7.33 [03-07-18 9:30:12 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:13 pm] | ssl/fog.csr: No need to sync fog.csr file to IRECFOG01 [03-07-18 9:30:13 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.28.25.165 [03-07-18 9:30:13 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:14 pm] | ssl/fog.csr: No need to sync fog.csr file to IREMEFOG01 [03-07-18 9:30:14 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.24.25.165 [03-07-18 9:30:14 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:16 pm] | ssl/fog.csr: No need to sync fog.csr file to IRGMSFOG01 [03-07-18 9:30:16 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.25.25.165 [03-07-18 9:30:16 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:17 pm] | ssl/fog.csr: No need to sync fog.csr file to IRGWCFOG01 [03-07-18 9:30:17 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.26.25.165 [03-07-18 9:30:17 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:17 pm] | ssl/fog.csr: No need to sync fog.csr file to IRHTEFOG01 [03-07-18 9:30:17 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.31.25.165 [03-07-18 9:30:17 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:18 pm] | ssl/fog.csr: No need to sync fog.csr file to IRJCEFOG01 [03-07-18 9:30:18 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.29.25.165 [03-07-18 9:30:18 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:19 pm] | ssl/fog.csr: No need to sync fog.csr file to IRLBEFOG01 [03-07-18 9:30:19 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.32.25.165 [03-07-18 9:30:19 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:20 pm] | ssl/fog.csr: No need to sync fog.csr file to IRLNEFOG01 [03-07-18 9:30:20 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.22.25.165 [03-07-18 9:30:20 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:20 pm] | ssl/fog.csr: No need to sync fog.csr file to IRMMSFOG01 [03-07-18 9:30:20 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.21.25.165 [03-07-18 9:30:20 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:21 pm] | ssl/fog.csr: No need to sync fog.csr file to IRNGEFOG01 [03-07-18 9:30:21 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.46.25.165 [03-07-18 9:30:21 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:22 pm] | ssl/fog.csr: No need to sync fog.csr file to IRPSEFOG01 [03-07-18 9:30:22 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.30.25.165 [03-07-18 9:30:22 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:23 pm] | ssl/fog.csr: No need to sync fog.csr file to IRSCHSFOG01 [03-07-18 9:30:23 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.39.25.165 [03-07-18 9:30:23 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:23 pm] | ssl/fog.csr: No need to sync fog.csr file to IRSMSFOG01 [03-07-18 9:30:23 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "fog.csr" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl" "/opt/fog/snapins/ssl"; exit' -u fog,[Protected] 10.27.25.165 [03-07-18 9:30:23 pm] * Started sync for Snapin ssl/fog.csr [03-07-18 9:30:23 pm] * Found Snapin to transfer to 14 nodes [03-07-18 9:30:23 pm] | File Name: ssl/CA [03-07-18 9:30:24 pm] * Starting Sync Actions [03-07-18 9:30:24 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.40.7.33 [03-07-18 9:30:24 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:24 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.28.25.165 [03-07-18 9:30:24 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:24 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.24.25.165 [03-07-18 9:30:24 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:25 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.25.25.165 [03-07-18 9:30:25 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:25 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.26.25.165 [03-07-18 9:30:25 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:25 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.31.25.165 [03-07-18 9:30:25 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:26 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.29.25.165 [03-07-18 9:30:26 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:26 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.32.25.165 [03-07-18 9:30:26 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:26 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.22.25.165 [03-07-18 9:30:26 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:27 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.21.25.165 [03-07-18 9:30:27 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:27 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.46.25.165 [03-07-18 9:30:27 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:27 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.30.25.165 [03-07-18 9:30:27 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:28 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.39.25.165 [03-07-18 9:30:28 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:28 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep..transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins/ssl/CA" "/opt/fog/snapins/ssl/CA"; exit' -u fog,[Protected] 10.27.25.165 [03-07-18 9:30:28 pm] * Started sync for Snapin ssl/CA [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: Cisco Jabber [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: Dell 3020 [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: Dell GetBootOrder [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: DESAA Secure Browser 10.0 [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: DisableRecovery [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: Java 8-151 [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: Ladibug [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: NAF Secure Browser 2.1.1.0 [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: Office16 [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: SEP 14 [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: Sketch up Pro 2017 [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: Software-Hardware Inventory [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - EME [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - GMS [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - GWC [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - IRHS [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - JCE [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - MMS [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - NGE [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - PSE [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - SCHS [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - SDSA [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA - SMS [03-07-18 9:30:28 pm] | There are no other members to sync to. [03-07-18 9:30:28 pm] * Not syncing Snapin between groups [03-07-18 9:30:28 pm] | Snapin Name: WIDA -LNE [03-07-18 9:30:29 pm] | There are no other members to sync to. [03-07-18 9:30:29 pm] * Not syncing Snapin between groups [03-07-18 9:30:29 pm] | Snapin Name: WIDA- LBE [03-07-18 9:30:29 pm] | There are no other members to sync to. [03-07-18 9:30:29 pm] * Not syncing Snapin between groups [03-07-18 9:30:29 pm] | Snapin Name: Z - Win10-GuestLogin [03-07-18 9:30:29 pm] | There are no other members to sync to. [03-07-18 9:30:29 pm] * Attempting to perform Group -> Nodes snapin replication. [03-07-18 9:30:29 pm] * Found Snapin to transfer to 14 nodes [03-07-18 9:30:29 pm] | Snapin Name: Cisco Jabber [03-07-18 9:30:29 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to 10.40.7.33 [03-07-18 9:30:29 pm] * Starting Sync Actions [03-07-18 9:30:29 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.10.40.7.33.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.40.7.33 [03-07-18 9:30:29 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:30:35 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRECFOG01 [03-07-18 9:30:35 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRECFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.28.25.165 [03-07-18 9:30:35 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:30:40 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IREMEFOG01 [03-07-18 9:30:40 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IREMEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.24.25.165 [03-07-18 9:30:40 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:30:42 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGMSFOG01 [03-07-18 9:30:42 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.25.25.165 [03-07-18 9:30:42 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:30:47 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRGWCFOG01 [03-07-18 9:30:47 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRGWCFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.26.25.165 [03-07-18 9:30:47 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:30:48 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRHTEFOG01 [03-07-18 9:30:48 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRHTEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.31.25.165 [03-07-18 9:30:48 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:30:53 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRJCEFOG01 [03-07-18 9:30:53 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRJCEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.29.25.165 [03-07-18 9:30:53 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:30:59 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLBEFOG01 [03-07-18 9:30:59 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLBEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.32.25.165 [03-07-18 9:30:59 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:31:04 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLNEFOG01 [03-07-18 9:31:04 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.22.25.165 [03-07-18 9:31:04 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:31:09 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRMMSFOG01 [03-07-18 9:31:09 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRMMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.21.25.165 [03-07-18 9:31:09 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:31:11 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRNGEFOG01 [03-07-18 9:31:11 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRNGEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.46.25.165 [03-07-18 9:31:11 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:31:16 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRPSEFOG01 [03-07-18 9:31:16 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRPSEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.30.25.165 [03-07-18 9:31:16 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:31:18 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRSCHSFOG01 [03-07-18 9:31:18 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSCHSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.39.25.165 [03-07-18 9:31:18 pm] * Started sync for Snapin Cisco Jabber [03-07-18 9:31:23 pm] | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRSMSFOG01 [03-07-18 9:31:23 pm] | CMD: lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.27.25.165 [03-07-18 9:31:23 pm] * Started sync for Snapin Cisco Jabber``` -
It may or may not be related but I ran into the dashboard and storage pages taking minutes to load yesterday. We could get to the login screen and start the login process but the dashboard was taking up to 4 minutes to load. We have seen a slight slowness in these screens since updating to 1.5.0 but not like this. We have 13 storage nodes and one happened to be down yesterday which I believe that the disk usage graph on the dashboard was waiting for a response from the node that was down. I disabled all the graphs for everything except the main server and the dashboard loads just as quick as the other pages now. The storage page was still loading very slow so I deleted the storage node that was down and that page again loads, slower than the rest but in a reasonable amount of time.
Is it possible that there is a timeout check somewhere for storage nodes that could be set lower, and is it possible that the traffic issue is related in some way waiting for a response that never comes?
-
@dclark Just found this by accident as I have been working on the replication code a fair bit. You might want to update to the latest
dev-branch. I have fixed all known replication issues in the last weeks.Important:
All nodes need to be updated at the same time or images will be replicated over and over again as some of the hashing code needed to be changed. Please make sure to stop replication on the master first (systemctl stop FOGImageReplicator; systemctl stop FOGSnapinReplicator), then update the storage node(s) and then update master node as a last step. -
@Sebastian-Roth I have updated the entire network to the 1.5.5.1 Development Branch tonight. So far things seem to be stable and showing much less network traffic between the server and storage nodes than I was showing previously. There was a 10-15mb outgoing heartbeat from each node to the server when replication was enabled that does not seem to be present now.
My only other concern would be the update to the kernel to solve the partition deletion issue during imaging that was taking so long, I had reverted back to the earlier version when using 1.5.0. Looks like it was updated in 1.5.5 but I will not be able to verify until tomorrow when I get into work.
-
@dclark said in Increased network traffic since 1.5.0 upgrade (Snapins):
Looks like it was updated in 1.5.5
Yeah, we fixed that!
-
@Sebastian-Roth Went through the day today without any reported issues and network activity has been at a minimum with the FOG communications between the server and nodes. Great Work Guys!
-
@dclark good you went to the dev-branch as I have fixed a snapin replication problem there which we had in the 1.5.5 release…
As you have a fair amount of storage nodes you might want to test the updates I will push to the
dev-branchfairly soon?! Will improve the web UI speed use even less traffic between the nodes. -
@Sebastian-Roth Just let me know when you have it a point you are comfortable with trying it and I will certainly deploy the latest development branch.
-
@dclark I just merged the stuff into
dev-branchan hour ago. Give it a try and see what you think of it. -
@Sebastian-Roth I have updated everything to the latest development version without issue tonight. The web gui definitely seems faster. I will watch for any oddities tomorrow while the system is in use. Thank You Guys again for all you do!
