Multicast just hangs

[rogalskij]

@george1421 Yes, I have the “em1” interface of the FOG server, and the network card of the Dell Computer I am trying to image on the same subnet.

The output of the “ip addr show” command on the FOG server is:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d4:ae:52:af:b5:63 brd ff:ff:ff:ff:ff:ff
    inet 150.155.1.70/20 brd 150.155.15.255 scope global noprefixroute dynamic em1
       valid_lft 704372sec preferred_lft 704372sec
    inet6 fe80::3d39:c85:7bf0:e61e/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether d4:ae:52:af:b5:64 brd ff:ff:ff:ff:ff:ff

[george1421]

@rogalskij ok now that we understand your hardware setup a bit more. To my question, have you ever imaged using your “BaseImage” image using unicast?

[rogalskij]

@george1421 Yes, when I capture or deploy an image using Unicast, everything is happy ducky wonderful. Images capture and deploy without an issue what so ever.

[george1421]

@rogalskij ok so here is where we are:

1. Its not the image because it deploys correctly using unicast
2. We know the installed network adapters and em1 is the correct network adapter, it has an ip address and is currently up
3. The ps command shows that udp-sender should be using network interface em1
4. The target computers and fog server is on the same vlan so no additional infrastructure work is needed.
5. At least some of the multicasts are getting through since the clients are able to check in and the stream starts.
6. It appears to hang at the partclone screen

We still don’t know if the infrastructure is setup correctly for multicasting (i.e. igmp snooping is enabled on vlan 1).
We don’t know if the multicast settings are right in the fog configuration.
We don’t know if the fog server’s firewall has been enabled but multicasts not allowed.

[rogalskij]

@george1421

Some additional information and questions:

I enabled “igmp snooping” on all of the switches and I verified that it is enabled on the switch that lab full of computers sits under.

I am happy to review the multicast settings. I put them in the main body of this post, do they look correct?

How do I check the firewall on the fog server (CentOS 7). I am pretty sure I disabled it entirely but can’t remember.

I did a config restore from my dev system which was a virtual machine. Could this be screwing something up? Something brought over incorrectly?

[george1421]

@rogalskij What network switches do you use?

I’ve been looking back in my docs and I found a documented multicasting issue with meraki switches.

[rogalskij]

@george1421 We use Cisco C2960S switches in those labs. The core is also Cisco.

[george1421]

@rogalskij Well then, lets assume the fog server is setup correctly. The firewall thing may not be an issue because the prerequisites for installing fog is the firewall being off. Some organizations, that isn’t allow so its turned on with specific rules fog needs to operate. If the multicast is not part of those rules then that function will be disabled. FWIW: systemctl stop firewalld and systemctl disable firewalld is what you need to stop and then disable the linux firewall.

So if you plug 2 clients into the same switch is the fog server and then schedule a multicast deployment job with max clients of 2, when that second client comes online does the multicast move forward? Now this is on the same switch as the FOG server.

Oh one other comment I found is to make sure you have port-fast or one of the other fast spanning tree protocols enabled on the switch.

[Sebastian Roth]

@rogalskij Sometimes old tasks can cause issues where partclone hangs on that screen. May I ask you to cancel all current tasks, reboot the FOG server and then schedule a fresh multicast task lets say for three machines (all on the same switch!). Let us know if the clients hang again.

If they do I ask you to run ps aux | grep sender again and post output here.

[george1421]

@Sebastian-Roth from a previous post:

Additionally, the output of the command you specified “sudo ps aux|grep udp-sender” is:

root 13864 0.0 0.0 115300 1480 ? S Aug30 0:00 sh -c /usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 1200 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p1.img;/usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 10 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p2.img;
root 14393 0.0 0.0 8688 660 ? S Aug30 0:00 /usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 10 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p2.img
root 31094 0.0 0.0 112708 992 pts/0 S+ 11:39 0:00 grep --color=auto udp-sender

There appears to be some stale multicast tasks running since 30-Aug.

[rogalskij]

@george1421 After a reboot there is no change. The computer is still deploying images via unicast without an issue. I updated the Kernel to the latest version, no change. One question, could I have the interfaces for multicast set wrong? Is there a way to check on the CentOS server what they are really named? Also inside the system can I check this? Just in case the WebUI is reporting it back incorrect?

[george1421]

@rogalskij said in Multicast just hangs:

@george1421 One question, could I have the interfaces for multicast set wrong? Is there a way to check on the CentOS server what they are really named?

We have already proved this out via the ip addr show what network address / adapters are in play here.

Both Sebastian and I recommended to start with just 2 systems on the same switch as the fog server.

[Sebastian Roth]

@rogalskij If your new test with two hosts hangs again, may I ask you to run ps aux | grep sender again and post output here. I want to make sure it start with the correct parameters.

[Sebastian Roth]

@rogalskij As well I wonder if you’ve gone through our testing guide on multicast?! https://wiki.fogproject.org/wiki/index.php/Troubleshoot_Downloading_-_Multicast#Testing_Multicast

[rogalskij]

@Sebastian-Roth I will be testing again today. My apologies on this taking so long. I will report back.

[rogalskij]

Ok more developments, we found when running the udpcast commands to test, the tests failed. I switched the Poweredge server (where FOG is installed) over to the same Cisco 2960S switch as the target computers and multicast worked perfectly!!! So it seems to be something with my Cisco 6509E core switch. I checked to make sure IGMP snooping was enabled on the core, but other than that I am unsure what to check. Any thoughts?

[Tom Elliott]

@rogalskij Maybe try setting up the multicast-rendevous with your FOG IP Server? You can use the TCP / IP, or use the Multicast that matches your TCP address.

[rogalskij]

@Tom-Elliott Good thinking, I just attempted that but it seemed to make no difference. My clients still seem to hang on the partclone screen. I did reach out to Cisco to check to see if my 6509E core switch has all the correct settings on it for multicasting. I also made sure the port the server was on is using “port-fast”. It worked like a charm the moment I plugged it into the edge switch. I will do some more testing while I wait for Cisco to answer me back. Sorry for all the back and forth with this, I really do appreciate this product, it’s developers, and the dedicated community behind it.

[rogalskij]

Found the issue! After some research and discussion with Cisco, we had to add “PIM” to the vlan on our core, even though both the server and client are both on the same vlan!

Used the command - ip pim sparse-dense-mode on vlan 1 interface and it started working like a charm! I really appreciate everyone’s assistance here. This will help our institution so very much.