Multicast Issues on Centos 7

Tom Elliott

@BedCruncher What’s output if you go to (in the browser)

http://ip.of.fog.here/fog/service/ipxe/boot.php?mac=macofhostwithcolons

Where macofhostwithcolons is the mac address (with colons) of the host that’s setup to do this tasking?

BedCruncher

@Tom-Elliott
Here’s the info. I also compared it to a second host that I had added to a group for multitasking, and it appears identical, barring the mac address of course. Let me know what else you can recommend and I’ll be glad to accommodate.

#!ipxe
set fog-ip 192.168.240.10
set fog-webroot fog
set boot-url http://${fog-ip}/${fog-webroot}
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.240.10/fog/ consoleblank=0 rootfstype=ext4 rootfstype=ext4 mac=00:26:b9:aa:14:ec ftp=192.168.240.10 storage=192.168.240.10:/images/ storageip=192.168.240.10 web=192.168.240.10/fog/ osid=5 consoleblank=0 irqpoll hostname=0026b9aa14ec chkdsk=0 img=W7Px64PreSysprep imgType=mps imgPartitionType=all imgid=1 imgFormat= PIGZ_COMP=-6 hostearly=1 port=50028 type=down mc=yes
imgfetch init_32.xz
boot

Tom Elliott

@BedCruncher When the client boots to begin the tasking, is it on a separate subnet from the 192.168.240.10 server?

BedCruncher

@Tom-Elliott
No it’s not. I have the em2 interface controlling the DHCP server and have a cable running to a 5 port Gig switch. It’s a dumb switch with no special routing or anything. I even checked under the dhcpd.leases file and can find it there.

lease 192.168.240.30 {
  starts 4 2016/04/28 12:50:28;
  ends 4 2016/04/28 18:50:28;
  cltt 4 2016/04/28 12:50:28;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:26:b9:aa:14:ec;
  uid "\001\000&\271\252\024\354";
}

EDIT:
Updated to build 7410 this morning to hopefully see if it might have been fixed.
EDIT 2:
Updated and initiated a new test and still getting the same results.

BedCruncher

Hey guys, I had to step away over the weekend. Has there been any new updates?

Sebastian Roth

@BedCruncher Now that we have the logging back can you please try running udp-sender by hand again using the exact same command as seen in the logs?

BedCruncher

@Sebastian-Roth
So I tried both of the udp-sender commands

 udp-sender --file /opt/fog/.fogsettings --log /tmp/multicast.log  --ttl 32 --nopointopoint --interface em2

which results in

13:41:14.387632 Using mcast address 232.168.240.10
13:41:14.387698 UDP sender for /opt/fog/.fogsettings at 192.168.240.10 on em2 
13:41:14.387708 Broadcasting control to 224.0.0.1

displaying in the logs and then no task found

and also

 cat /images/W7Px64PreSysprep/d1p1.img | /usr/local/sbin/udp-sender --interface em2 --min-receivers 2 --max-wait 600 --portbase 50028 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/W7Px64PreSysprep/d1p2.img | /usr/local/sbin/udp-sender --interface em2 --min-receivers 2 --max-wait 600 --portbase 50028 --full-duplex --ttl 32 --nokbd --nopointopoint;

which results in the same message as above, but not the repeated no active task found entries. I also updated to build 7470 earlier. It is acting differently. I don’t see that the udp-sender and udp-receiver are talking any more.

Sebastian Roth

@BedCruncher When using this command you need to bring up at least two clients with udp-receive to actually make it start sending - see the --min-receivers 2 option:

cat /images/W7Px64PreSysprep/d1p1.img | /usr/local/sbin/udp-sender --interface em2 --min-receivers 2 --max-wait 600 --portbase 50028 --full-duplex --ttl 32 --nokbd --nopointopoint

So please try this again by hand. Start the udp-sender up on your FOG server like this (don’t need the logging option - will just print on screen). Then boot two of your clients in debug mode and run: udp-receiver --file test.img --portbase 50028 --ttl 32 --nokbd (on both clients). If your network is setup properly this should work!!!

BedCruncher

@Sebastian-Roth
I did that through my dumb switch and also removed the min-receivers switch and tried it manually. Still didn’t work. I then plugged the server directly into the switch and tried only the simplified command that I got to work before and it also failed.

I will try that tomorrow as I won’t have access to the server until then.

Tom Elliott

@BedCruncher Can you run a join.me session?

Tom Elliott

@BedCruncher Seeing as, as I understand it, you’ve removed managed switch scenarios as a potential causing problems, but you’re still having issues, does your FOG server (or something else) have a firewall running that’s just blocking any multicast/UDP traffic?

Wayne Workman

@Wayne-Workman said in Multicast Issues on Centos 7:

Look at this:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_Downloading_-_Multicast

https://wiki.fogproject.org/wiki/index.php?title=Multicast

let us know how it goes, we are here to help.

Many times, I’ll post things and later on an answer is found that was already in one of the links I posted… seeing how long this thread has gone for, I’m just going to repost these links to be looked at again by the OP and any future readers.

BedCruncher

@Tom-Elliott

To answer your questions, I will handle the second one first. I went through and verified that yes, I do have a firewall on my server (firewalld), and as a way to get around the default zone of public that the em2 interface was assigned, I moved it to the “trusted” zone so that it wouldn’t filter any traffic whatsoever. After that, I then was able to get some more information for muticasting testing for you.

I got to playing around with the the commands and debug mode on an individual client and I noticed a slight issue when comparing the multicast port in the fog settings with the command that is logged when I trigger and try to run the multicast task. The logged part I am referencing is --portbase 50028 but when I checked that against what I had listed in the fogsettings multicast port 56904. I attempted to change to settings to mirror what is in the log file, but continually fail at the part where I attempt to run the command

cat /images/W7Px64PreSysprep/d1p1.img | /usr/local/sbin/udp-sender --interface em2 --min-receivers 2 --max-wait 600 --portbase 50028 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/W7Px64PreSysprep/d1p2.img | /usr/local/sbin/udp-sender --interface em2 --min-receivers 2 --max-wait 600 --portbase 50028 --full-duplex --ttl 32 --nokbd --nopointopoint;

However, if I follow that part near the bottom middle of this Multicasting page, I can get it to work running the same command

gunzip -c /images/W7Px64PreSysprep/d1p1.img | /usr/local/sbin/udp-sender --interface em2 --min-receivers 2 --max-wait 600 --portbase 50028 --full-duplex --ttl 32 --nokbd --nopointopoint;gunzip -c /images/W7Px64PreSysprep/d1p2.img | /usr/local/sbin/udp-sender --interface em2 --min-receivers 2 --max-wait 600 --portbase 50028 --full-duplex --ttl 32 --nokbd --nopointopoint;

Then both clients begin to receive the images as expected.

BedCruncher

@Wayne-Workman

I followed up as I just posted on the second link and with some tweaking I was able to get somewhere on this, I believe you should be able to understand what I’ve done according to the second link and it’s testing steps.

Tom Elliott

@BedCruncher The troubleshooting guide forgets that the client actually does the gunzip action, not the server. So your finding makes perfect sense. To fix the client test, you first need to pull the request to a gunzip equiv as well.

For example,

The receiver command should be:

mkfifo /tmp/pigz1

udp-receiver --nokbd --portbase 50028 --ttl 32 --mcast-rdv-address <FOGSERVERIP> 2>/dev/null >/tmp/pigz1 &
pigz -d -c </tmp/pigz1 | partclone.restore --ignore_crc -O /dev/sda2

Of course adjust the /dev/sda to the respective file you’re trying to load.

Potentially, the most important part is that of the --mcast-rdv-address as it’s what tells the client WHERE to get the broadcast stream.

Sebastian Roth

@BedCruncher Definitely shouldn’t make a different if you are using compressed or uncompressed image data (when testing)!!! Sure pushing the image to disk is a different story but just trying to get the multicast thing running it should work either way I suppose.

Tom Elliott

@Sebastian-Roth If he’s testing real data (d1p1.img for example) it most definitely will.

BedCruncher

@Tom-Elliott
I will try the command you posted above and report back what it finds. Thanks.

BedCruncher

@Tom-Elliott
Ok, so now we are getting somewhere. I had to run another multicast test really quickly on both of the client computers as I had earlier blown away the partition tables that were restored. That was just to have them properly put back in place. I since have ran the command that you specified on the client and am testing it for both of the stored disk images and they are restoring back. I will let it run it’s course… only take 15 minutes or less for the restore to take place and then try doing the multicast test directly and see if it hangs or not.

EDIT:
The manual restore for both partitions that I did worked correctly and both now boot. I will try scheduling a multicast task again and see where we are at.

BedCruncher

OK!!! I think we have something here. So I finally figured out that part of the problem seems to be in the firewall zone as specified in the Centos 7 Setup guide. The issue I think stems from that the public zone seems to be blocking the multicast ports. I even explicitly told the firewalld daemon to add the interface em2 to the trusted zone, but it never did unless I manually specified in the NIC interface file ZONE=trusted. This seems to be a bug of sorts in that project.

I suppose that you could also specify a port range for the firewalld daemon to allow through. In my case the NIC hosting the FOG Server is segregated my other network. So in this case I don’t care to have all ports open on that interface because there is no reason to block that traffic. Keep in mind the interface em1 is still in the public zone and more locked down and restricted.

The result seems to be that I am now able get it to consistently image across at least two devices and so far it has persisted across device and server reboots. So I think we have made a good leap in that regards. I will keep checking back here for a few days to try and update if I run into issues.

Thank you all for persistently fighting with me to get this rolling. I do much appreciate all you have done for me and with me.
@Sebastian-Roth @Wayne-Workman @Tom-Elliott

Multicast Issues on Centos 7

139

11.7k

17.2k

154.7k