High network usage while idle (no tasks)
-
I recently updated our FOG server to 1.5.9, running on CentOS 7, no OS update. After the FOG update, I noticed about 200 Mbps of network traffic outbound on the switch port that our server is connected to. This traffic is present even when there are no tasks scheduled in FOG.
Some packet captures revealed that what I was seeing appears to be TCP traffic from the FOG server to “random” client machines. The traffic has persisted across multiple days and sits right around 200 Mbps consistently. Reboots of the server have no effect, the traffic starts up again immediately after the machine boots.
Any ideas as to what may be happening? Thank you!
-
@kenm Well, 25 MB/s is and isn’t really that much.
With that said the for server does do limited reaching out to the computers listed in inventory to see if they are up or down. The fog server does this by attempting to connect to port 445 on the target computer. It will only do this for computers in inventory. If you have a storage node in your environment it will also connect to the storage node over http then ftp to replicate image files. Beyond that I can’t think of any reason the fog server (or fog code specifically) would be generating traffic on your network. If you have concerns I would surely setup wireshark on a mirrored port or use tcpdump on the FOG server to have it capture all traffic the fog server generates. If you need help with the tcpdump command I can help. Then review the tcpdump command in wireshark looking for unexpected traffic.
-
It’s a lot for this server, which, prior to the upgrade, wasn’t associated with much more than 1/10 of this traffic at idle.
A 10-second tcpdump on the FOG server yields nearly 70k TCP 80 packets, nothing on port 445. I can’t read much in the packets, but I catch fragments of what looks like GNU GPL here and there. I looked at a handful of TCP conversations, and they all start out with a “HTTP GET /fog/client/SmartInstaller.exe” request, followed by tons of TCP ACKs.
I just checked the Client Updater settings and it’s disabled. We’ve never had clients set to update. Is there another setting that has been changed somewhere during the upgrade that could cause this behavior?
-
@kenm OK so the first thing I would look at is who are the actors and who is starting the conversation (sorry worked in secops for quite a while). Is the FOG server talking with any foreign systems (foreign as in not part of your network)? The smart installer requests almost seems like they should be coming from clients workstation asking if there is any updated fog client programs.
If you want to run the tcpdump again but filter on the mac address of the FOG server this post that file to a file share site and IM me the link I’ll take a look at it. That is ok if you don’t too, but I’m kind of interesting in what the FOG server is doing. The interesting points are (again) who the actors are, who is starting the conversation, and what ports are being used (ports will kind of give you the clue to what service is being used).
-
@george1421 I don’t see any contact outside of our network. Each conversation is initiated by the client machine, and the smart installer requests are coming from the client to the FOG server. Everything is over port 80, exclusively.
Following the TCP stream in Wireshark (10.10.56.21 is FOG):
… followed by illegible characters for the most part.I can get a capture file to you in a few hours once I’m back at my desk.
-
@kenm Ok we might want to get the @Developers involved here because I don’t know and or use the FOG Client program. But it would be helpful to know the version of FOG Client on one of these chatty workstations, then we can compare it with the version on the FOG server (still talking about the fog client). Its possible that the FOG Clients are programmed to detect an update to the FOG client and try to download and install it (guess). With something not working right in the process it keeps querying and asking for that file over and over again (guess).
As long as you’ve identified the traffic, I don’t need to see it.
-
@kenm FOG Client version went from 0.11.19 to 0.12.0 I believe.
The FOG Client does auto update when it can. I suspect this is part of the “chatter” you’re seeing. Though I’m more keen to know if the fog client is even running properly on the machines.
Understand, September was an expiration of a code signing certificate for 0.11.19 and a new one was created for 0.12.0, but we’ve seen some issues with the client auto updating. It appears like it cannot see the FOG Server CA, and seems to be fixed by uninstalling the client and reinstalling it.
Not sure how much more help can be gleaned from this though.
-
@tom-elliott Thanks. I’ll try installing the new client and see if the issue stops.
On a side note, I’ve used FOG for a number of years to help image/manage a couple thousand machines, and I’ve always been very pleased with it. Support from the mods and developers appears to always be top-notch and timely. I want everyone to know that the work is appreciated.
-
@KenM I am sure it’s the fog-client trying to update but fails in every loop because of the run out code signing cert. Re-installing the new version should fix that.
