Imaging Causes Phone Problems.

[george1421]

@matthewk2010 Ok so you have a hosted PBX.

The unicast / multicast thing… You have to specifically do a multicast deployment. This is where you would deploy to 1 or more machines at the same exact time. You have to specifically setup a multicast session. If you don’t specifically know you are doing this, or just picking imaging then you are doing a unicast imaging.

[matthewk2010]

@george1421

I believe I am using multicast. I only see multicast settings when I am looking at the FOG Configurations and to my knowledge I have not enabled unicast. Is multicast the default?

We have around 4000 devices that are connected the network, but only 1500 of those would need access to the FOG Server.

[george1421]

@matthewk2010 OK still driving towards a multicasting answer here.

How to you image a computer? Do you pxe boot the computer, go into the fog menu and select register and image?

Do you schedule an imaging task on the FOG server then pxe boot the target computer and it images right away?

Its probably not relevant but I want to put half of the issues out of my mind when considering this issue.

[matthewk2010]

@george1421

I appreciate your help so ask anything you need to.

Basically all I do is manually boot the PC to the PXE Menu and select deploy image. I select the image I want and then it starts. I have only done this one PC at a time.

[george1421]

@matthewk2010 OK good, you are doing a unicast image deployment.

OK now lets focus on the device that connects your vlans? What are you using as your router? Does that router also interface with your ISP connection to your office PBX?

[Jaymes Driver]

@matthewk2010 This is unicast.

Multicast is when you send the same image to multiple computers at the same time as a “group” task.

They often wait in a Queue and download the same information at the same time. (they will all wait at 30% until all clients are are 30% and then proceed).

[matthewk2010]

@george1421

We are using a layer 3 CISCO 3750X IOS 12.2(58) SE2 for routing and yes it is our connection to our ISP.

[george1421]

@matthewk2010 My bet is that the unicast imaging is flooding your router (between the vlans) and that is causing your audio issues.

Do you have QoS setup on your router (not the ISP router) but the router between the vlans? What we need to have happen is the vlan router needs to put the RTP (audio) part of the voip call ahead of all other traffic.

[Jaymes Driver]

@george1421 Ahh yes, lovely QoS

[george1421]

@george1421 We can test this theory out of you can move a target computer to the same vlan as the FOG server, but have it physically located where your other target computers are.

What I’m getting at is to have the target computer and FOG server on the same vlan then image. See if that causes your VoIP issues. If it doesn’t then we can focus on your router. My guess it will image fine without any voip issues.

[matthewk2010]

Okay I will give that a try tomorrow and see what happens.

Here are the QOS settings that are currently on the routing switch.

mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos srr-queue input bandwidth 70 30
mls qos srr-queue input threshold 1 80 90
mls qos srr-queue input priority-queue 2 bandwidth 30
mls qos srr-queue input cos-map queue 1 threshold 2 3
mls qos srr-queue input cos-map queue 1 threshold 3 6 7
mls qos srr-queue input cos-map queue 2 threshold 1 4
mls qos srr-queue input dscp-map queue 1 threshold 2 24
mls qos srr-queue input dscp-map queue 1 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue input dscp-map queue 1 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue input dscp-map queue 2 threshold 3 32 33 40 41 42 43 44 45
mls qos srr-queue input dscp-map queue 2 threshold 3 46 47
mls qos srr-queue output cos-map queue 1 threshold 3 4 5
mls qos srr-queue output cos-map queue 2 threshold 1 2
mls qos srr-queue output cos-map queue 2 threshold 2 3
mls qos srr-queue output cos-map queue 2 threshold 3 6 7
mls qos srr-queue output cos-map queue 3 threshold 3 0
mls qos srr-queue output cos-map queue 4 threshold 3 1
mls qos srr-queue output dscp-map queue 1 threshold 3 32 33 40 41 42 43 44 45
mls qos srr-queue output dscp-map queue 1 threshold 3 46 47
mls qos srr-queue output dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23
mls qos srr-queue output dscp-map queue 2 threshold 1 26 27 28 29 30 31 34 35
mls qos srr-queue output dscp-map queue 2 threshold 1 36 37 38 39
mls qos srr-queue output dscp-map queue 2 threshold 2 24
mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue output dscp-map queue 3 threshold 3 0 1 2 3 4 5 6 7
mls qos srr-queue output dscp-map queue 4 threshold 1 8 9 11 13 15
mls qos srr-queue output dscp-map queue 4 threshold 2 10 12 14
mls qos queue-set output 1 threshold 1 100 100 50 200
mls qos queue-set output 1 threshold 2 125 125 100 400
mls qos queue-set output 1 threshold 3 100 100 100 400
mls qos queue-set output 1 threshold 4 60 150 50 200
mls qos queue-set output 1 buffers 15 25 40 20
mls qos

[george1421]

@matthewk2010 well I’m not a cisco freek so it will take me until tomorrow to decode this. I am a very old network engineer so I understand the bits and bytes of the issue.

The other thing is that you need to ensure your phones are tagging their traffic with using dscp (according to what you just posted).

[Wayne Workman]

@Jaymes-Driver said in Imaging Causes Phone Problems.:

Ahh yes, lovely QoS

Crap QoS. IPv4 itself is flawed. A large transfer doesn’t allow other packets through, it’s like a missile going down the wire. There’s been an integration recently into the Linux Kernel called fq_codl that solves this issue, it was authored by Dave Taht. If you used routers and switches that run using a new linux kernel, you can blast the network as heavy as you want to with as many streams as you want to, as many massive file transfers and imaging as you want to - with no configured QoS - and no seperation of services via VLans - and VoIP will be crystal clear - because the kernel will make room for the smaller packets to go through no matter what.

If you need QoS, then really you need a new network.

</rant>

[george1421]

@george1421 Well I looked at the code and now I know why I’m not a cisco engineer. IMO it looks like the queues are backwards. BUT doing a little research I see that your code is text book example of how to setup QoS on a cisco. ref: https://mrncciew.com/2013/02/24/best-practice-qos-config/

I’m still of a mind that if QoS is setup correctly no amount of “other traffic” should disrupt your voip calls. So assuming that QoS is setup correctly then I would have to ask you a bit more about your environment.

1. Do you have the computers daisy chained through the phones to the building switch?
2. Is your entire network GbE?
3. When you are imaging, do all phone has issues with call quality or is it just in the network area where the target computer is?
4. Tell me about your switch to switch uplinks, do you run just a single link between the switches or do you have LAG groups setup between your IDF switches and your MDF (core) switch?

I can say from FOG’s point of view, its job is to move images from the FOG server to the target system as fast as it can, with the only restrictions of how fast the target system can download the image from the FOG server and expand it to the local disk. Its totally possible for a FOG server and a modern client to saturate a 100Mb/s link. On my campus I get about 6.7GB/m transfer rates. That translates to about 111MB/s (that is a bit misleading because that also includes the expansion rate of the image at the target) but its in the same ballpark. The theoretical maximum transfer rate of a 1 GbE network is 125MB/s. So playing loose with the numbers, right now I can saturate a 1 GbE network just deploying an image.

[george1421]

@Wayne-Workman said in Imaging Causes Phone Problems.:

If you need QoS, then really you need a new network.

While I know that was a rant, that not really true. QoS is used to guaranty a certain level of service. RTP (the audio stream) is timing critical, packets arriving out of order or late will cause audio quality issues. that is why you should setup QoS even on a non-saturated network. This give your audio stream first chance at the switch port’s transmit buffer, saturated or not. As I said with my fog server and modern clients, I can almost saturate a 1 GbE network with just one unicast stream. Deploying 2 images to machines in the same network area would put it over the edge if certain precautions where not taken.

[Wayne Workman]

@george1421 said in Imaging Causes Phone Problems.:

Deploying 2 images to machines in the same network area would put it over the edge if certain precautions where not taken.

Because your switches and routers do not use fq_codl. This fundamentally changes how traffic handling is done.

Cisco is even aware of the problem that fq_codl solves. Cisco’s solution - which is yet to be completed - is called pie something or another.

[matthewk2010]

Okay, so I moved the PC I was trying to image to the same VLAN as the FOG server and there is no phone issue. So do you think if I change some QOS around in my switch to prioritize phone traffic it will fix the issue?

I am going to try and reach out to our rep from the company that manages some of our core switches. Hopefully he can provide some info on the current QOS settings. I will post what I hear from him.

Thanks for your help,
Matt

[george1421]

@matthewk2010

I’m not sure I can explain this with a post but here goes.

For an accurate test the machine that was being imaged and caused phone issues. Is there a way to move it only to the same vlan as the fog server.

What is important here is 2 points.

1. The network switch to network switch data path between the fog server and the PC that was causing the issue must remain consistent.
2. By changing the target system to the same VLAN as the FOG server, you are bypassing the router between the vlans, but keeping the same data path.

So what’s not clear in my mind. Is the issue caused by your router not being fast enough, or having the proper QoS rules in place.

OR, you have a bottle neck in the communication path between the FOG server and the target system that is causing the problem.

IF all you change is the vlan association with the target computer, does the voip communications exist.

If you can answer that you should be able to narrow in on the problem.

I can tell you one other thing. QoS should be an end to end solution. Meaning that all switches on your campus should have QoS setup and defined, not just your router. Because any link between the FOG server and the target computer can become saturated causing the communications problems.

[matthewk2010]

Sorry, this took so long to post, but I have found the fix. The issue was that we had storm-control broadcast, storm-control unicast, and storm-control multicast levels set to low on our core switch (layer 3 switch where the routing happens). I removed these and all is well. I appreciate all the help you all supplied.