Imaging Causes Phone Problems.

[matthewk2010]

@george1421

We are using a layer 3 CISCO 3750X IOS 12.2(58) SE2 for routing and yes it is our connection to our ISP.

[george1421]

@matthewk2010 My bet is that the unicast imaging is flooding your router (between the vlans) and that is causing your audio issues.

Do you have QoS setup on your router (not the ISP router) but the router between the vlans? What we need to have happen is the vlan router needs to put the RTP (audio) part of the voip call ahead of all other traffic.

[Jaymes Driver]

@george1421 Ahh yes, lovely QoS

[george1421]

@george1421 We can test this theory out of you can move a target computer to the same vlan as the FOG server, but have it physically located where your other target computers are.

What I’m getting at is to have the target computer and FOG server on the same vlan then image. See if that causes your VoIP issues. If it doesn’t then we can focus on your router. My guess it will image fine without any voip issues.

[matthewk2010]

Okay I will give that a try tomorrow and see what happens.

Here are the QOS settings that are currently on the routing switch.

mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos srr-queue input bandwidth 70 30
mls qos srr-queue input threshold 1 80 90
mls qos srr-queue input priority-queue 2 bandwidth 30
mls qos srr-queue input cos-map queue 1 threshold 2 3
mls qos srr-queue input cos-map queue 1 threshold 3 6 7
mls qos srr-queue input cos-map queue 2 threshold 1 4
mls qos srr-queue input dscp-map queue 1 threshold 2 24
mls qos srr-queue input dscp-map queue 1 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue input dscp-map queue 1 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue input dscp-map queue 2 threshold 3 32 33 40 41 42 43 44 45
mls qos srr-queue input dscp-map queue 2 threshold 3 46 47
mls qos srr-queue output cos-map queue 1 threshold 3 4 5
mls qos srr-queue output cos-map queue 2 threshold 1 2
mls qos srr-queue output cos-map queue 2 threshold 2 3
mls qos srr-queue output cos-map queue 2 threshold 3 6 7
mls qos srr-queue output cos-map queue 3 threshold 3 0
mls qos srr-queue output cos-map queue 4 threshold 3 1
mls qos srr-queue output dscp-map queue 1 threshold 3 32 33 40 41 42 43 44 45
mls qos srr-queue output dscp-map queue 1 threshold 3 46 47
mls qos srr-queue output dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23
mls qos srr-queue output dscp-map queue 2 threshold 1 26 27 28 29 30 31 34 35
mls qos srr-queue output dscp-map queue 2 threshold 1 36 37 38 39
mls qos srr-queue output dscp-map queue 2 threshold 2 24
mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue output dscp-map queue 3 threshold 3 0 1 2 3 4 5 6 7
mls qos srr-queue output dscp-map queue 4 threshold 1 8 9 11 13 15
mls qos srr-queue output dscp-map queue 4 threshold 2 10 12 14
mls qos queue-set output 1 threshold 1 100 100 50 200
mls qos queue-set output 1 threshold 2 125 125 100 400
mls qos queue-set output 1 threshold 3 100 100 100 400
mls qos queue-set output 1 threshold 4 60 150 50 200
mls qos queue-set output 1 buffers 15 25 40 20
mls qos

[george1421]

@matthewk2010 well I’m not a cisco freek so it will take me until tomorrow to decode this. I am a very old network engineer so I understand the bits and bytes of the issue.

The other thing is that you need to ensure your phones are tagging their traffic with using dscp (according to what you just posted).

[Wayne Workman]

@Jaymes-Driver said in Imaging Causes Phone Problems.:

Ahh yes, lovely QoS

Crap QoS. IPv4 itself is flawed. A large transfer doesn’t allow other packets through, it’s like a missile going down the wire. There’s been an integration recently into the Linux Kernel called fq_codl that solves this issue, it was authored by Dave Taht. If you used routers and switches that run using a new linux kernel, you can blast the network as heavy as you want to with as many streams as you want to, as many massive file transfers and imaging as you want to - with no configured QoS - and no seperation of services via VLans - and VoIP will be crystal clear - because the kernel will make room for the smaller packets to go through no matter what.

If you need QoS, then really you need a new network.

</rant>

[george1421]

@george1421 Well I looked at the code and now I know why I’m not a cisco engineer. IMO it looks like the queues are backwards. BUT doing a little research I see that your code is text book example of how to setup QoS on a cisco. ref: https://mrncciew.com/2013/02/24/best-practice-qos-config/

I’m still of a mind that if QoS is setup correctly no amount of “other traffic” should disrupt your voip calls. So assuming that QoS is setup correctly then I would have to ask you a bit more about your environment.

1. Do you have the computers daisy chained through the phones to the building switch?
2. Is your entire network GbE?
3. When you are imaging, do all phone has issues with call quality or is it just in the network area where the target computer is?
4. Tell me about your switch to switch uplinks, do you run just a single link between the switches or do you have LAG groups setup between your IDF switches and your MDF (core) switch?

I can say from FOG’s point of view, its job is to move images from the FOG server to the target system as fast as it can, with the only restrictions of how fast the target system can download the image from the FOG server and expand it to the local disk. Its totally possible for a FOG server and a modern client to saturate a 100Mb/s link. On my campus I get about 6.7GB/m transfer rates. That translates to about 111MB/s (that is a bit misleading because that also includes the expansion rate of the image at the target) but its in the same ballpark. The theoretical maximum transfer rate of a 1 GbE network is 125MB/s. So playing loose with the numbers, right now I can saturate a 1 GbE network just deploying an image.

[george1421]

@Wayne-Workman said in Imaging Causes Phone Problems.:

If you need QoS, then really you need a new network.

While I know that was a rant, that not really true. QoS is used to guaranty a certain level of service. RTP (the audio stream) is timing critical, packets arriving out of order or late will cause audio quality issues. that is why you should setup QoS even on a non-saturated network. This give your audio stream first chance at the switch port’s transmit buffer, saturated or not. As I said with my fog server and modern clients, I can almost saturate a 1 GbE network with just one unicast stream. Deploying 2 images to machines in the same network area would put it over the edge if certain precautions where not taken.

[Wayne Workman]

@george1421 said in Imaging Causes Phone Problems.:

Deploying 2 images to machines in the same network area would put it over the edge if certain precautions where not taken.

Because your switches and routers do not use fq_codl. This fundamentally changes how traffic handling is done.

Cisco is even aware of the problem that fq_codl solves. Cisco’s solution - which is yet to be completed - is called pie something or another.

[matthewk2010]

Okay, so I moved the PC I was trying to image to the same VLAN as the FOG server and there is no phone issue. So do you think if I change some QOS around in my switch to prioritize phone traffic it will fix the issue?

I am going to try and reach out to our rep from the company that manages some of our core switches. Hopefully he can provide some info on the current QOS settings. I will post what I hear from him.

Thanks for your help,
Matt

[george1421]

@matthewk2010

I’m not sure I can explain this with a post but here goes.

For an accurate test the machine that was being imaged and caused phone issues. Is there a way to move it only to the same vlan as the fog server.

What is important here is 2 points.

1. The network switch to network switch data path between the fog server and the PC that was causing the issue must remain consistent.
2. By changing the target system to the same VLAN as the FOG server, you are bypassing the router between the vlans, but keeping the same data path.

So what’s not clear in my mind. Is the issue caused by your router not being fast enough, or having the proper QoS rules in place.

OR, you have a bottle neck in the communication path between the FOG server and the target system that is causing the problem.

IF all you change is the vlan association with the target computer, does the voip communications exist.

If you can answer that you should be able to narrow in on the problem.

I can tell you one other thing. QoS should be an end to end solution. Meaning that all switches on your campus should have QoS setup and defined, not just your router. Because any link between the FOG server and the target computer can become saturated causing the communications problems.

[matthewk2010]

Sorry, this took so long to post, but I have found the fix. The issue was that we had storm-control broadcast, storm-control unicast, and storm-control multicast levels set to low on our core switch (layer 3 switch where the routing happens). I removed these and all is well. I appreciate all the help you all supplied.