Multicast just hangs
rogalskij last edited by rogalskij
I have most things working correctly in FOG. One thing that is still not working properly is multicasting. When I attempt it with a known working image, the computers reboot just fine but then they get to the “partclone” screen and they never start imaging. I noticed in the “multicast” settings that my interface was some odd designation. I tried changing it to either “eth0” or “em1” to match the CentOS server settings but to no avail. Not sure what I am doing wrong. Can anyone assist? My desktop support team would love to image our computer labs with this on campus.
UDPCAST INTERFACE eth0
UDPCAST STARTINGPORT 62142
MULTICAST MAX SESSIONS 64
UDPCAST MAXWAIT 20
MULTICAST PORT OVERRIDE 0
MULTICAST DUPLEX FULL_DUPLEX
This is the screen it gets stuck on:
Found the issue! After some research and discussion with Cisco, we had to add “PIM” to the vlan on our core, even though both the server and client are both on the same vlan!
Used the command - ip pim sparse-dense-mode on vlan 1 interface and it started working like a charm! I really appreciate everyone’s assistance here. This will help our institution so very much.
@Tom-Elliott Good thinking, I just attempted that but it seemed to make no difference. My clients still seem to hang on the partclone screen. I did reach out to Cisco to check to see if my 6509E core switch has all the correct settings on it for multicasting. I also made sure the port the server was on is using “port-fast”. It worked like a charm the moment I plugged it into the edge switch. I will do some more testing while I wait for Cisco to answer me back. Sorry for all the back and forth with this, I really do appreciate this product, it’s developers, and the dedicated community behind it.
@rogalskij Maybe try setting up the multicast-rendevous with your FOG IP Server? You can use the TCP / IP, or use the Multicast that matches your TCP address.
Ok more developments, we found when running the udpcast commands to test, the tests failed. I switched the Poweredge server (where FOG is installed) over to the same Cisco 2960S switch as the target computers and multicast worked perfectly!!! So it seems to be something with my Cisco 6509E core switch. I checked to make sure IGMP snooping was enabled on the core, but other than that I am unsure what to check. Any thoughts?
@Sebastian-Roth I will be testing again today. My apologies on this taking so long. I will report back.
@rogalskij As well I wonder if you’ve gone through our testing guide on multicast?! https://wiki.fogproject.org/wiki/index.php/Troubleshoot_Downloading_-_Multicast#Testing_Multicast
@rogalskij If your new test with two hosts hangs again, may I ask you to run
ps aux | grep senderagain and post output here. I want to make sure it start with the correct parameters.
@george1421 One question, could I have the interfaces for multicast set wrong? Is there a way to check on the CentOS server what they are really named?
We have already proved this out via the
ip addr showwhat network address / adapters are in play here.
Both Sebastian and I recommended to start with just 2 systems on the same switch as the fog server.
@george1421 After a reboot there is no change. The computer is still deploying images via unicast without an issue. I updated the Kernel to the latest version, no change. One question, could I have the interfaces for multicast set wrong? Is there a way to check on the CentOS server what they are really named? Also inside the system can I check this? Just in case the WebUI is reporting it back incorrect?
@Sebastian-Roth from a previous post:
Additionally, the output of the command you specified “sudo ps aux|grep udp-sender” is: root 13864 0.0 0.0 115300 1480 ? S Aug30 0:00 sh -c /usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 1200 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p1.img;/usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 10 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p2.img; root 14393 0.0 0.0 8688 660 ? S Aug30 0:00 /usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 10 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p2.img root 31094 0.0 0.0 112708 992 pts/0 S+ 11:39 0:00 grep --color=auto udp-sender
There appears to be some stale multicast tasks running since 30-Aug.
@rogalskij Sometimes old tasks can cause issues where partclone hangs on that screen. May I ask you to cancel all current tasks, reboot the FOG server and then schedule a fresh multicast task lets say for three machines (all on the same switch!). Let us know if the clients hang again.
If they do I ask you to run
ps aux | grep senderagain and post output here.
@rogalskij Well then, lets assume the fog server is setup correctly. The firewall thing may not be an issue because the prerequisites for installing fog is the firewall being off. Some organizations, that isn’t allow so its turned on with specific rules fog needs to operate. If the multicast is not part of those rules then that function will be disabled. FWIW:
systemctl stop firewalldand
systemctl disable firewalldis what you need to stop and then disable the linux firewall.
So if you plug 2 clients into the same switch is the fog server and then schedule a multicast deployment job with max clients of 2, when that second client comes online does the multicast move forward? Now this is on the same switch as the FOG server.
Oh one other comment I found is to make sure you have port-fast or one of the other fast spanning tree protocols enabled on the switch.
@george1421 We use Cisco C2960S switches in those labs. The core is also Cisco.
@rogalskij What network switches do you use?
I’ve been looking back in my docs and I found a documented multicasting issue with meraki switches.
Some additional information and questions:
I enabled “igmp snooping” on all of the switches and I verified that it is enabled on the switch that lab full of computers sits under.
I am happy to review the multicast settings. I put them in the main body of this post, do they look correct?
How do I check the firewall on the fog server (CentOS 7). I am pretty sure I disabled it entirely but can’t remember.
I did a config restore from my dev system which was a virtual machine. Could this be screwing something up? Something brought over incorrectly?
@rogalskij ok so here is where we are:
- Its not the image because it deploys correctly using unicast
- We know the installed network adapters and em1 is the correct network adapter, it has an ip address and is currently up
- The ps command shows that udp-sender should be using network interface em1
- The target computers and fog server is on the same vlan so no additional infrastructure work is needed.
- At least some of the multicasts are getting through since the clients are able to check in and the stream starts.
- It appears to hang at the partclone screen
We still don’t know if the infrastructure is setup correctly for multicasting (i.e. igmp snooping is enabled on vlan 1).
We don’t know if the multicast settings are right in the fog configuration.
We don’t know if the fog server’s firewall has been enabled but multicasts not allowed.
@george1421 Yes, when I capture or deploy an image using Unicast, everything is happy ducky wonderful. Images capture and deploy without an issue what so ever.
@rogalskij ok now that we understand your hardware setup a bit more. To my question, have you ever imaged using your “BaseImage” image using unicast?
rogalskij last edited by Sebastian Roth
@george1421 Yes, I have the “em1” interface of the FOG server, and the network card of the Dell Computer I am trying to image on the same subnet.
The output of the “ip addr show” command on the FOG server is:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether d4:ae:52:af:b5:63 brd ff:ff:ff:ff:ff:ff inet 220.127.116.11/20 brd 18.104.22.168 scope global noprefixroute dynamic em1 valid_lft 704372sec preferred_lft 704372sec inet6 fe80::3d39:c85:7bf0:e61e/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether d4:ae:52:af:b5:64 brd ff:ff:ff:ff:ff:ff