MultiCast very slow

[Wayne Workman]

What version of FOG?

Remember multicast runs at the speed of the slowest member in the group. It could be a bad patch cable or a bad HDD slowing the whole group down.

Just as a baseline, can you see what speeds you get when you image an individual computer in the same group?

I played around with doing an iPerf throughput task type for FOG in the past - might be time to resurrect that project.

[plegrand]

@Wayne-Workman
Hello
Version 6237 in the cloud
The deploy task is not finished (4 hours for the moment)
I will make the unicast test just after
Thanks for your help

[Junkhacker]

if multicast is going that slow, it would probably go much faster to just unicast instead

[plegrand]

@Junkhacker
I would like to use multicast and solve this problem.
Could it be a version multicast problem?
I configure my switch with version 3

[Wayne Workman]

@plegrand We can’t really troubleshoot while the multicast is running. I’d suggest waiting until it’s done, and then we can troubleshoot with just one or two computers.

[plegrand]

@Wayne-Workman
I’ll made the test tomorrow with one station and unicast
then with 2 stations and i’ll give you the result

[plegrand]

@Wayne-Workman
I just made the test unicast (deploy and not multicast)
It’s a room with 12 stations.
2 stations failed
10 works fine (1.6 GB/min)
i’ll tell you more tomorrow

[plegrand]

@Wayne-Workman
Hello,
as i said yesterday, once multicast finished, i launched an unicast task on the 12 stations.
it worked fine. I thought 2 stations were failed but it was because of the limit of unicast clients (10)
Then where is it possible to change the limit of unicast ?
Fog Configuration => General settings => FOG_QUEUESIZE
or
Storage Management => Default Member => Max Clients

For the multicast problem here is the result :
for 12 station
Multicast => 04:40:00
Unicast => 00:28:00

Thanks for your help

[Wayne Workman]

@plegrand said:

Storage Management => Default Member => Max Clients

That’s the correct spot, but I’d strongly recommend decreasing it to about 3 instead of increasing it further. I see far better performance when less are running simultaneously.

Now, just to see if multicast is the issue, can you multicast to just two computers? Make a temp group, put two computers in it, and then use that group to multicast. Is there a speed difference?

If it’s a failing hdd or faulty patch cable, and that’s one of the two you choose, it’d be difficult to analyze the results.

Also - the switch is important when multicasting.

There is a certain amount of processing power involved with replicating a packet to all ports - and cheap switches just don’t cut it.

There’s also maximum total throughput to consider. For example, at home I have a consumer grade Cisco Small business switch. It’s 1Gbps on each port and has 5 ports. But total internal throughput is 3Gbps. That means that I would never be able to multicast at home using that switch at 1Gbps speeds for more than 2 computers at a time. However I have a new 8 port 1Gbps z-link switch from China (for 28 bucks new) that has internal throughput of 5Gbps. Meaning that device would be able to multicast to 4 computers at once with 1Gbps speed to each.

Again, cheap equipment just doesn’t cut it when it comes to multicast and really needing every port to operate at it’s maximum speed. The higher end Cisco equipment usually doesn’t have a problem though with this.

[EAHarvey]

I would be willing to bet that it has a lot to do with your compression level. Check under FOG Configuration>FOG Settings>FOG Boot Settings>FOG_PIGZ_COMP. Set this to 2 or 3 and see if that makes a big difference. You may need to redo your image that your pushing but I don’t think you will. This may help[ with the speed issue.

[Wayne Workman]

@EAHarvey The compression setting is only used for image upload/capture. He would need to re-upload to try a different compression setting.

[EAHarvey]

@Wayne-Workman Roger that. I was thinking he might would have re-upload but wasn’t sure. Thank you sir.

[plegrand]

Now, just to see if multicast is the issue, can you multicast to just two computers? Make a temp group, put two computers in it, and then use that group to multicast. Is there a speed difference?

I’ll make the test

Also - the switch is important when multicasting.

It 's Alcatel switch OS6250 and OS6450 configured like that : http://plegrand1.free.fr/pasted_image001.png
I’m not sure if the configuration is ok
Thanks for your help

[plegrand]

@Wayne-Workman Hello
just for information :
On my switch i configure (force) the igmp version to “3”, may be i should use the default version : “2” ?

An other question : i thought that fog used “udpcast” for multicasting, is it true ?

[Wayne Workman]

@plegrand said:

On my switch i configure (force) the igmp version to “3”, may be i should use the default version : “2” ?

Try it and let us know. at this point, anything that causes a change is good.

An other question : i thought that fog used “udpcast” for multicasting, is it true ?

I think fog uses udpsender.

[JJ Fullmer]

@plegrand A couple questions
What are the specs (cpu(s) model and speed, Ram speed and size, virtual/physical) of your FOG server?
What are the specs of the workstations (specifically the slowest one)?
What is the compression rating on the image you’re pushing?

Something to understand about multicast (and someone should correct me if I’m wrong) is that is does all the decompression on the server and then sends the packets out following multicast protocol. The idea of multicast is to do more on the server side and less on the client and get a constant speed. It also is meant to prevent using a file or part of a file that is in use to make a fail safe against corrupting files and such. The downside is that if you’re using an image with a compression level of 9 and your server isn’t all that powerful, multi-cast is going to slow down a lot compared to unicast where decompression is done on the client.

My point is if your server has less power or less optimization for decompressing then your clients, multicast will just be slower. Considering your speed difference of multicast and unicast, my guess would be this is the case.

Also, a side note in favor of just using unicast, I have used fog to deploy to a lab of 40 computers (10 at a time, which can be changed with the Max clients setting in the storage configuration, just fyi) with the same image on unicast and had no problems with corruption and the speed was glorious.

Another limiting factor is client and server port speeds. If your server is on a 10/100Mb port, you should upgrade that because it will make your life painful. But if your server is on a Gigabit switch then 10 clients on 100Mb switches will all run at around 1.7 GB/min (estimated mathematically and based on experience) If everything is on gigabit switches/ports then you might see a decrease in client speed as you add more simultaneous clients , but will still be around 2 or 3 times faster for 10 clients. Just to give you a little reference off the top of my head.

It would probably be a good idea to start recording average speeds for common configurations and post them in the forum somewhere so people have a reference of how fast to configuration ought to be performing.

[Wayne Workman]

@Arrowhead-IT said:

Something to understand about multicast (and someone should correct me if I’m wrong) is that is does all the decompression on the server and then sends the packets out following multicast protocol.

That’s wrong, multicast sends the data compressed just like unicast deployments do. Decompression happens on the client side. And the idea of multicast is to broadcast throughout a broadcast domain, in order to send data only once instead of many times. There is a lot of overhead with multicast, but the more clients that are receiving the stream, the bigger the payoff.

Don’t feel bad - I thought the same thing, but tom corrected me then as I am now for you. The post is somewhere on here - from maybe a year ago lol.

[plegrand]

@Arrowhead-IT
lscpu
Architecture : x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Boutisme : Little Endian
Processeur(s) : 8
Liste de processeur(s) en ligne : 0-7
Thread(s) par cœur : 2
Cœur(s) par socket : 4
Socket(s) : 1
Nœud(s) NUMA : 1
Identifiant constructeur : GenuineIntel
Famille de processeur : 6
Modèle : 30
Nom de modèle : Intel Xeon CPU X3480 @ 3.07GHz
Révision : 5
Vitesse du processeur en MHz : 3059.238
BogoMIPS : 6118.47
Virtualisation : VT-x
Cache L1d : 32K
Cache L1i : 32K
Cache L2 : 256K
Cache L3 : 8192K
Nœud NUMA 0 de processeur(s) : 0-7

free -m
total used free shared buffers cached
Mem: 32233 1192 31040 23 196 624
-/+ buffers/cache: 372 31860
Swap: 65514 0 65514

df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/sda1 8,2G 1,2G 6,6G 15% /
udev 10M 0 10M 0% /dev
tmpfs 6,3G 8,6M 6,3G 1% /run
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda5 2,7G 445M 2,1G 18% /var
/dev/sda7 360M 3,8M 334M 2% /tmp
/dev/sda8 842G 25G 774G 4% /home

eth0: NIC Copper Link is Up, 1000 Mbps full duplex
eth1: NIC Copper Link is Up, 1000 Mbps full duplex

I made a test : Multicast and unicast 12 stations

**Multicast **=> 04h 40min
**Unicast ** => 28min

[JJ Fullmer]

@Wayne-Workman I wonder if I got compression part from that post a year ago. Some of that is what I remember from my college Networking class.

@plegrand Looks like the problem shouldn’t be the server power, and I was wrong about that part anyway.
Considering the speed difference you already are seeing, why not just use unicast? I’m just curious if there’s a specific requirement for multicast?

[Sebastian Roth]

@plegrand Any news on this? Do you still see slow speeds when using multicast?