SVN 2979 multicast issues
-
Also, the storage nodes don’t need to be anything fancy (I’ve said this before?)
An old dual-core with a gigabit interface will get the job done in an acceptable amount of time.
Here, in my environment, FOG is virtualized on one of our servers, and can blast an image out to 1 client in 8 minutes (almost 50GB image, uncompressed). We can image 29 systems with that particular image in 29 minutes using multicast (I blame a bad patch cable somewhere).
-
[quote=“Wayne Workman, post: 42508, member: 28155”]…
The problem is that they are within different broadcast domains.
…[/quote]Not correct from my point of view. WOL is a bit of an issue if you want to wake up clients in another subnet. But Multicasting (IF setup correctly) sould work beyond broadcast domains! I have my server in 192.168.6.x/24 and some of my clients in 192.168.6.x/24 (VLAN6) but also others in 192.168.5.x/24 (VLAN5). All doing multicast imaging without any issue!!!
-
I know I had broken the multicast to other bc domains for a little while and I don’t remember when it was found and corrected for. That said I know this is corrected for in current version of svn. I’d recommend updating and see if that helps you out. For wol I’d say install the wolbroadcast plugin as wol across vlans is what this was designed for. You will need your main routing switch or router set to enable ip-directed-broadcasts
-
[quote=“jamesb, post: 42509, member: 27742”]So I’m basically going to need to set up a storage node for each vlan we have in order for multicast to work in all of our buildings?[/quote]
I stand somewhat corrected.
I found this: [url]http://networkengineering.stackexchange.com/questions/10065/how-broadcasting-works-on-different-networks[/url]
Which basically says that limited broadcasts are always dropped, but a router can be configured to allow directed broadcasts by running [B][SIZE=14px][FONT=Droid Sans Mono][COLOR=#222222]ip directed-broadcast[/COLOR][/FONT][/SIZE][/B] (I assume on a Cisco router).
Have you done this, jamesb ?
-
I haven’t tried the ip directed-broadcast yet, I’m having our network person get that setup up on our switches and routers now.
-
Before my network guy wants to open up the switches with this setting he wants to try a different mcast address. I’ve tried to get the 239 address to be used but FOG always uses the 236 address. I’ve overwritten in the FOG multicast settings in the web GUI but it’s not showing the change in the log files.
-
As I already said… From my point of view multicast is NOT broadcast! You don’t need directed broadcasts to make multicast work. Just my two cents…
Do you still have errors in your apache log file??Have you gone through this guide yet? [url]http://fogproject.org/wiki/index.php?title=Troubleshooting_a_multicast[/url]
-
[quote=“jamesb, post: 42582, member: 27742”]Before my network guy wants to open up the switches with this setting he wants to try a different mcast address. I’ve tried to get the 239 address to be used but FOG always uses the 236 address. I’ve overwritten in the FOG multicast settings in the web GUI but it’s not showing the change in the log files.[/quote]
SVN 2920 & up supports custom multicast address and port settings, along with a few other settings too I think.
That change was made to get FOG working at my location, and it works fine here. We don’t use the default [U]broadcast address[/U] for our broadcast domains, there’s custom setup intended to somewhat contain any multicast stream to prevent network-slowdown.
I’m not sure how the setup on the router & switches was accomplished here (not my job), I just know that it works.
239 & 236 addresses? I don’t follow.
By default, according to the IP info you gave earlier, the broadcast address for the segment that the FOG server is on should be: 172.28.2.255
Have you gotten multicast to work on the FOG server’s V-lan yet? What are the errors associated with [U]that[/U]?
-
Just my 2 cents on Multicasting issues. I use fog to image two labs at SCSU. Multicast never would work. Last week was spring break I had the time to run a couple of tests. I followed all the instructions to test multicast on the forums but multicast deploy would hang on the blue screen and that was that. I do not have administrative access to the production switches.
Images are complex with ubunutu with many VMs in the images. Image type; RAW, compression is set to 3. Client hardware; HP Thin Clients with 8 GB of memory and 126GB SSD.
Fog Server Dell 6GB. I do have a HP on the way in a month that will have 64GB to test at the end of spring semester so I may be able to speed thing up a bit.
I decided to swap out, (our office of information technology Cisco managed switches) with Cisco SBM switches out of the “box”.
Multicast in Fog works just fine.I tested the following:
Multicast 8 HP Thin Clients on Cisco SMB Switches - average image 22 minutes.
Multicast 15 HP Thin Clients on Cisco SMB Switches - average image 26 minutes.
Unicast 8 HP Thin Clients on Cisco SMB Switches - average image 1hr 26 minutes.
Unicast 15 HP Thin Clients on Cisco SMB Switches - average image 2hr 54 minutes.Unicast to 15 Thin Clients on production switches to 15 systems would vary from 2 to 4 hours.
I do multi partition non resizable NTFS 40GB images and this averages 24 minutes unicast. I did not have time to test this in multicast.
So multicast does work and for larger deployments is much faster. Performance is 23 minutes vs 1hr and 26 minutes. I have sent the fog logs to our OIT group to have them look into their Cisco’s. I am planning to replace the current production switches . I would suggest to add a line to the testing of multicast to just swap out any old switches.
-
I think one of the biggest core-problems with I.T. support people multicasting is being unable to access their switches / routers.
These dumb barriers between network teams and I.T. support are created. A technician will say “Multicast isn’t working”, network admin says “It’s set up, you must not be doing it right”… vicious circle. And that’s if they even BOTHER to help you… Either person could be wrong, honestly. But, without one being able to see the other’s setup (or communicate really well), it’s pretty pointless to even try to troubleshoot.
I despise things like this.
-
[quote=“Wayne Workman, post: 44225, member: 28155”]I think one of the biggest core-problems with I.T. support people multicasting is being unable to access their switches / routers.
These dumb barriers between network teams and I.T. support are created. A technician will say “Multicast isn’t working”, network admin says “It’s set up, you must not be doing it right”… vicious circle. And that’s if they even BOTHER to help you… Either person could be wrong, honestly. But, without one being able to see the other’s setup (or communicate really well), it’s pretty pointless to even try to troubleshoot.
I despise things like this.[/quote]
After being told by our network group that multicast was enabled on the production switches on Monday, we found out yesterday that only one of the two switches supports layer three. The ports in the labs are randomly patched into the switches, no documentation. Wire rack is locked, etc…
Ran another test this morning. 15 thin clients using production switches. Start the multicast, all systems pxe and sitting on gray screen. Multicast not starting. Go into the task list the ones in the group that are ready are patched to level three multicast. The ones that are still connecting were patched to the layer two switch. Made bets with network guys. Killed task. Created a new group with thin client members that were waiting in the first job (six computers). Started multicast, executes perfectly. I’m very lucky to have had the fog server randomly patched into the switch with layer thee capabilities. I had given up on multicast last summer.
Four years I’ve been using FOG now, always hear it from the networking group that there is nothing wrong. But they put in all into emails so now it is documented. Getting my own switches this summer.
126GB in 23 minutes is just shy of the 20 minutes (the fastest it could go), the only limitation for the multicast was the GB network cards.
Now if I can figure out the TFTP Boot errors…
-
[quote=“Steven B, post: 44546, member: 24174”]After being told by our network group that multicast was enabled on the production switches on Monday, we found out yesterday that only one of the two switches supports layer three. The ports in the labs are randomly patched into the switches, no documentation. Wire rack is locked, etc…
Ran another test this morning. 15 thin clients using production switches. Start the multicast, all systems pxe and sitting on gray screen. Multicast not starting. Go into the task list the ones in the group that are ready are patched to level three multicast. The ones that are still connecting were patched to the layer two switch. Made bets with network guys. Killed task. Created a new group with thin client members that were waiting in the first job (six computers). Started multicast, executes perfectly. I’m very lucky to have had the fog server randomly patched into the switch with layer thee capabilities. I had given up on multicast last summer.
Four years I’ve been using FOG now, always hear it from the networking group that there is nothing wrong. But they put in all into emails so now it is documented. Getting my own switches this summer.
126GB in 23 minutes is just shy of the 20 minutes (the fastest it could go), the only limitation for the multicast was the GB network cards.
Now if I can figure out the TFTP Boot errors…[/quote]
Well, that’s a big jump from where you were before.
We should be able to help with the TFTP errors. I’ll read through this thread again and update this post if I think of anything.
Just read through the thread again. Didn’t see anything about TFTP.
You’ll need to post the errors you’re seeing. A picture of them would be fine, too. -
It’s been a while since I last posted about this but me and the network team has made some headway but we hit another wall. In FOG we had 2 machines in debug mode for the udp-receive and we were able to multicast a log file successfully. However whenever we tried an image file we would always the the result: Timeout notAnswered=[0,1] notReady=[0,1]
Does anybody know why we were able to do a log file but not an image file? -
Where was the log file located? Where on the FOG server?
Is it in a different place than /images ? -
It was the multicast.log file located in /opt/fog/log.
-
Make sure NFS service doesn’t have any errors…
[CODE]sudo service nfs-kernel-server status[/CODE]Make sure the FTP service doesn’t have any errors…
[CODE]sudo service vsftpd status[/CODE]You can restart the FTP service like this:
[CODE]sudo service vsftpd restart[/CODE]Check what’s allowed in/out of your firewall:
[CODE]iptables -L[/CODE]And, here’s some stuff Tom posted earlier here. I think you should try these commands again to see if it helps.
[CODE]sudo service FOGMulticastManager stop
sudo service FOGImageReplicator stop
sudo service FOGScheduler stop
sudo service FOGMulticastManager start
sudo service FOGImageReplicator start
sudo service FOGScheduler start[/CODE]#LetsMakeScripts
-
We have tried the stop and starting of the services many times with no real changes. Here is what my iptables list says:
Chain INPUT (policy ACCEPT)
target prot opt source destinationChain FORWARD (policy ACCEPT)
target prot opt source destinationChain OUTPUT (policy ACCEPT)
target prot opt source Destination -
Maybe upgrade to the latest SVN??
-
I just upgraded to 3180 and it still doesn’t work. Unicast works fine but is there some specific folder permissions that need to be in place for mulicast to work on the server. I doubt there is but I’m running out of ideas.
-
Here is exactly what we used to test the connections:
root@fogVM:/images/DellOptiplex3020syspreptest# udp-sender --file /opt/fog/log/multicast.log --ttl 32 --mcast-data-address 239.168.1.1 --min-receivers 2
Udp-sender 20120424
Using full duplex mode
UDP sender for /opt/fog/log/multicast.log at 172.28.2.21 on eth0
Broadcasting control to 224.0.0.1
New connection from 172.28.16.36 (#0) 00000009
Ready. Press any key to start sending data.
New connection from 172.28.16.49 (#1) 00000009
Ready. Press any key to start sending data.
Starting transfer: 00000009
bytes= 2 200 re-xmits=0000001 ( 50.0%) slice=0112 - 0
Transfer complete.
Disconnecting #0 (172.28.16.36)
Disconnecting #1 (172.28.16.49)Here is the one for the image:
root@fogVM:/images/DellOptiplex3020syspreptest# gunzip -c “/images/DellOptiplex3020syspreptest/d1p1.img” | /usr/local/sbin/udp-sender --min-receivers 2 --mcast-data-address 239.168.1.1 --portbase 9000 --interface 172.28.2.21 --full-duplex --ttl 32
Udp-sender 20120424
UDP sender for (stdin) at 172.28.2.21 on eth0
Broadcasting control to 224.0.0.1
New connection from 172.28.16.36 (#0) 00000009
Ready. Press any key to start sending data.
New connection from 172.28.16.49 (#1) 00000009
Ready. Press any key to start sending data.
Starting transfer: 00000009
Timeout notAnswered=[0,1] notReady=[0,1] nrAns=0 nrRead=0 nrPart=2 avg=6696
Timeout notAnswered=[0,1] notReady=[0,1] nrAns=0 nrRead=0 nrPart=2 avg=6696
Timeout notAnswered=[0,1] notReady=[0,1] nrAns=0 nrRead=0 nrPart=2 avg=6696
Timeout notAnswered=[0,1] notReady=[0,1] nrAns=0 nrRead=0 nrPart=2 avg=6696
