Multicast just hangs

george1421

@rogalskij fwiw the relevant lines in regards to which interface is here

Command: /usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 1200 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p1.img;/usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 10 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p2.img;

Task startedUdp-sender 20120424
Using mcast address 238.155.1.70
UDP sender for /images/BaseImage/d1p1.img at x.x.x.x on em1 
Broadcasting control to 224.0.0.1

george1421

So the first question I would have is: Are the target systems and the FOG server on the same subnet (vlan)?

rogalskij

@george1421 Yes, they are both on “vlan 1” for the moment. Both are on the same subnet.

Sebastian Roth

@rogalskij Is your VLAN 1 bound to em1 on your FOG server?

rogalskij

@Sebastian-Roth I checked the Apache log you mentioned, but all I see from that day doesn’t make a ton of sense to me:

[Fri Aug 30 15:25:11.052318 2019] [mpm_prefork:notice] [pid 2739] AH00170: caught SIGWINCH, shutting down gracefully
[Fri Aug 30 15:28:26.997884 2019] [core:notice] [pid 2727] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Aug 30 15:28:27.032137 2019] [suexec:notice] [pid 2727] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Aug 30 15:28:27.101535 2019] [lbmethod_heartbeat:notice] [pid 2727] AH02282: No slotmem from mod_heartmonitor
PHP Warning: Module ‘ldap’ already loaded in Unknown on line 0
[Fri Aug 30 15:28:27.268250 2019] [mpm_prefork:notice] [pid 2727] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.2.21 configured – resuming normal operations
[Fri Aug 30 15:28:27.268287 2019] [core:notice] [pid 2727] AH00094: Command line: ‘/usr/sbin/httpd -D FOREGROUND’

Does this make sense? Am I looking at something wrong here?

george1421

@rogalskij said in Multicast just hangs:

Yes, they are both on “vlan 1” for the moment. Both are on the same subnet.

Please clarify, you have 2 network adapters on the same subnet(vlan) do they have different IP address subnets? If they are on the same subnet, there may be an issue.

When you have a multicast running if you run the following command from the fog server command prompt: sudo ps aux|grep udp-sender you will see the current syntax that called the multicast sender.

So its getting all the way to partclone bits and then its failing? Can you/have you unicast this image before?

rogalskij

@george1421 Wait, you aren’t allowed to have the FOG server on the same subnet as the clients?! This is how we do most everything right now. We plan to subnet our devices later on, but previously with Ghost and other multicast products we just multicast with devices and the server on the same subnet. Is this still possible?

Additionally, the output of the command you specified “sudo ps aux|grep udp-sender” is:

root 13864 0.0 0.0 115300 1480 ? S Aug30 0:00 sh -c /usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 1200 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p1.img;/usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 10 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p2.img;
root 14393 0.0 0.0 8688 660 ? S Aug30 0:00 /usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 10 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p2.img
root 31094 0.0 0.0 112708 992 pts/0 S+ 11:39 0:00 grep --color=auto udp-sender

As you can see, it sees the interface em1, unless I am wrong and em1 isn’t the name of the interface, but that is what it says when I do an “ip addr” command on the server.

george1421

@rogalskij said in Multicast just hangs:

Wait, you aren’t allowed to have the FOG server on the same subnet as the clients?!

Just for clarity I read that you have 2 network interfaces on the same subnet. Is that accurate?

So from your output command I see you have 2 multicasts running at the moment from 30-Aug.

OK what do you get when you run the ip addr show command?

rogalskij

@george1421 Yes, I have the “em1” interface of the FOG server, and the network card of the Dell Computer I am trying to image on the same subnet.

The output of the “ip addr show” command on the FOG server is:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d4:ae:52:af:b5:63 brd ff:ff:ff:ff:ff:ff
    inet 150.155.1.70/20 brd 150.155.15.255 scope global noprefixroute dynamic em1
       valid_lft 704372sec preferred_lft 704372sec
    inet6 fe80::3d39:c85:7bf0:e61e/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether d4:ae:52:af:b5:64 brd ff:ff:ff:ff:ff:ff

george1421

@rogalskij ok now that we understand your hardware setup a bit more. To my question, have you ever imaged using your “BaseImage” image using unicast?

rogalskij

@george1421 Yes, when I capture or deploy an image using Unicast, everything is happy ducky wonderful. Images capture and deploy without an issue what so ever.

george1421

@rogalskij ok so here is where we are:

Its not the image because it deploys correctly using unicast
We know the installed network adapters and em1 is the correct network adapter, it has an ip address and is currently up
The ps command shows that udp-sender should be using network interface em1
The target computers and fog server is on the same vlan so no additional infrastructure work is needed.
At least some of the multicasts are getting through since the clients are able to check in and the stream starts.
It appears to hang at the partclone screen

We still don’t know if the infrastructure is setup correctly for multicasting (i.e. igmp snooping is enabled on vlan 1).
We don’t know if the multicast settings are right in the fog configuration.
We don’t know if the fog server’s firewall has been enabled but multicasts not allowed.

rogalskij

@george1421

Some additional information and questions:

I enabled “igmp snooping” on all of the switches and I verified that it is enabled on the switch that lab full of computers sits under.

I am happy to review the multicast settings. I put them in the main body of this post, do they look correct?

How do I check the firewall on the fog server (CentOS 7). I am pretty sure I disabled it entirely but can’t remember.

I did a config restore from my dev system which was a virtual machine. Could this be screwing something up? Something brought over incorrectly?

george1421

@rogalskij What network switches do you use?

I’ve been looking back in my docs and I found a documented multicasting issue with meraki switches.

rogalskij

@george1421 We use Cisco C2960S switches in those labs. The core is also Cisco.

george1421

@rogalskij Well then, lets assume the fog server is setup correctly. The firewall thing may not be an issue because the prerequisites for installing fog is the firewall being off. Some organizations, that isn’t allow so its turned on with specific rules fog needs to operate. If the multicast is not part of those rules then that function will be disabled. FWIW: systemctl stop firewalld and systemctl disable firewalld is what you need to stop and then disable the linux firewall.

So if you plug 2 clients into the same switch is the fog server and then schedule a multicast deployment job with max clients of 2, when that second client comes online does the multicast move forward? Now this is on the same switch as the FOG server.

Oh one other comment I found is to make sure you have port-fast or one of the other fast spanning tree protocols enabled on the switch.

Sebastian Roth

@rogalskij Sometimes old tasks can cause issues where partclone hangs on that screen. May I ask you to cancel all current tasks, reboot the FOG server and then schedule a fresh multicast task lets say for three machines (all on the same switch!). Let us know if the clients hang again.

If they do I ask you to run ps aux | grep sender again and post output here.

george1421

@Sebastian-Roth from a previous post:

Additionally, the output of the command you specified “sudo ps aux|grep udp-sender” is:

root 13864 0.0 0.0 115300 1480 ? S Aug30 0:00 sh -c /usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 1200 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p1.img;/usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 10 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p2.img;
root 14393 0.0 0.0 8688 660 ? S Aug30 0:00 /usr/local/sbin/udp-sender --interface em1 --min-receivers 3 --max-wait 10 --portbase 56590 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/BaseImage/d1p2.img
root 31094 0.0 0.0 112708 992 pts/0 S+ 11:39 0:00 grep --color=auto udp-sender

There appears to be some stale multicast tasks running since 30-Aug.

rogalskij

@george1421 After a reboot there is no change. The computer is still deploying images via unicast without an issue. I updated the Kernel to the latest version, no change. One question, could I have the interfaces for multicast set wrong? Is there a way to check on the CentOS server what they are really named? Also inside the system can I check this? Just in case the WebUI is reporting it back incorrect?

george1421

@rogalskij said in Multicast just hangs:

@george1421 One question, could I have the interfaces for multicast set wrong? Is there a way to check on the CentOS server what they are really named?

We have already proved this out via the ip addr show what network address / adapters are in play here.

Both Sebastian and I recommended to start with just 2 systems on the same switch as the fog server.

Multicast just hangs

258

12.0k

17.3k

155.2k