FOG 1.2.0 Multicast creates multiple single udpcast sessions

[Wayne Workman]

Simple question, do all the members of the group use the same image?

[SKasai]

Yes. We even tried doing a simple 2 computer test and the resultant was seeing both machines starting their own separate UDP cast session. Image source is the same. Example from the multicast log:

[03-20-15 4:50:45 pm] | Task (131) xxx56-57 is new!
[03-20-15 4:50:45 pm] | Task (131) /images/03132015 image file found.
[03-20-15 4:50:45 pm] | Task (131) 1 client(s) found.
[03-20-15 4:50:45 pm] | Task (131) xxx56-57 sending on base port: 51130
[03-20-15 4:50:45 pm] CMD: cat /images/03132015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 51130 --interface bond0 --mcast-data-address 239.xxx.xxx.241 --full-duplex --ttl 32 --nokbd;
[03-20-15 4:50:45 pm] | Task (131) xxx56-57 has started.
[03-20-15 4:50:56 pm] | Task (131) xxx56-57 is already running PID 32173
[03-20-15 4:50:56 pm] | Task (132) xxx56-57 is new!
[03-20-15 4:50:56 pm] | Task (132) /images/03132015 image file found.
[03-20-15 4:50:56 pm] | Task (132) 1 client(s) found.
[03-20-15 4:50:56 pm] | Task (132) xxx56-57 sending on base port: 60678
[03-20-15 4:50:56 pm] CMD: cat /images/03132015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 60678 --interface bond0 --mcast-data-address 239.xxx.xxx.241 --full-duplex --ttl 32 --nokbd;
[03-20-15 4:50:56 pm] | Task (132) xxx56-57 has started.

As you see here… This is one job, yet it generates two UDP sessions for 2 machines. This is not me spamming the task in succession. You will also notice it says ‘–min-receivers 1’… It should say ‘2’

[Tom Elliott]

What version of fog is this?

[SKasai]

This would be for FOG 1.2.0, running on CentOS 6.6

[Sebastian Roth]

How do you schedule the task? Group -> Basic Tasks -> Multicast?

[SKasai]

Group -> Search for Group -> Select Multicast -> Create Multicast Job.

[Wayne Workman]

What distribution of Linux and what version?

When was the last time this machine was re-booted?

[Sebastian Roth]

Has multicast ever worked before on this system. Or is this an issue ever since you installed this system?

[SKasai]

As mentioned above, running CentOS 6.6, using FOG version 1.2.0.

Multicast did work earlier, however, I believe, at one point we had to tweak something with regards to ipxe boot.php as we have one lab where it needs to ‘flip-flop’ between booting from one drive to another (sda - Linux, sdb - Windows), which the default fog ipxe did not allow at the time…

[Tom Elliott]

Are you sure you only edited the boot.php file? it seems somebody may have made edits to Host.class.php maybe?

[SKasai]

Well, looking at /opt/fog/service/common/lib…

We had to hack the MulticastTask.class.php for some reason… This is what we have:

[url=“/_imported_xf_attachments/1/1833_MulticastTask.class.php?:”]MulticastTask.class.php[/url]

[Tom Elliott]

If you’re running 1.2.0 that file should not be doing anything. What’s in the file /opt/fog/service/etc/config.php

[Sebastian Roth]

[quote=“SKasai, post: 44747, member: 29107”]Well, looking at /opt/fog/service/common/lib…

We had to hack the MulticastTask.class.php for some reason… This is what we have:[/quote]
Not sure from which version this file is coming (maybe Tom knows better) but it seems kinda old as the class MulticastTask does not extend FOGBase. Not sure if this plays a role but I definitely wonder if your installation is a bit mixed up with code from different versions and patched files on top of that. Wouldn’t wonder if things go wrong with this.

Do you have a spare machine (e.g. just a desktop machine) to setup a new FOG server (version 1.2.0 or current SVN if you like) to see if your multicast issues go away.

It’s very hard to guess what’s going wrong when you use a different set of code than we’d guess you have when talking about version 1.2.0.

[SKasai]

<?php

define( “WEBROOT”, “/var/www/html/fog” );
?>

[SKasai]

Sorry for taking so long to get back to everyone here… A lot of busy work and had to put this on the back burner…

Found the issue that was causing the problem. We are currently using cfengine and it pushed the 0.3.2 version of MulticastTask.class.php, which is completely not the right one for the 1.2.0, for obvious reasons, but was able to generate the right command with the wrong reasons.

Restoring the original MulticastTask.class.php that comes with 1.2.0 seems to have fixed it.

The reason we had cfengine pushing a modified version of the 0.3.2 config was due to needing to ‘hack’ the multicasttask.class.php to make it work a little easier with the Cisco switches. This was done prior to my time so I don’t know what it was exactly.

[SKasai]

Well, looks like I may have jumped the gun on that conclusion… A day later, the problem is back and still trying to figure out the issue…

But to put this in a little better context for all of you… Here is the hack on the file for /var/www/html/fog/lib/fog/MulticastTask.class.php that we were using… The First block is our hack, the Second block the Original

---

92,93c92,93

                          $cmd = 'cat '.$strRec.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';
                          $cmd .= 'cat '.$strSys.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';

---

                          $cmd = 'cat '.$strRec.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';
                          $cmd .= 'cat '.$strSys.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';

96c96

                          $cmd = 'cat '.$strSys.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';

---

                          $cmd = 'cat '.$strSys.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';

121c121

                                  $cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';

---

                                  $cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';

125c125

                          $cmd = 'cat '.rtrim($this->getImagePath(),'/').'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';

---

                          $cmd = 'cat '.rtrim($this->getImagePath(),'/').'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';

134c134

                          if ($handle = opendir($this->getImagePath()))

---

                          if($handle = opendir($this->getImagePath()))

153c153

                                  $cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';

---

                                  $cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';

178c178

                                  $cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';

---

                                  $cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';

---

So basically, we just added --mcast-data-address 239.x.x.x to force it to do the broadcast to the 239.x.x.x for the Cisco to allow it to work properly… What we are seeing, though is that when it creates command, it seems to do $this->getClientCount() as ‘1’ instead of the number of clients, but generates the clientcount’s # of processes.

[A Former User]

@SKasai So are you saying that multicasting was actually working recently after you thought you found the problem but now it stopped again? That could be a clue as to what it could maybe be.

[Tom Elliott]

You may be willing to try SVN/Trunk/GIT or whatever you want to call it of FOG. Development is what I try to call it, but I do refer often to trunk or svn as well.

It shouldn’t have the problems you’re seeing, and should work fairly well. I’m aware of a quirk or two but it seems to work fine.

Please give it a shot.

There’s also alot of added functionality.

[SKasai]

@isaiah658 said:

@SKasai So are you saying that multicasting was actually working recently after you thought you found the problem but now it stopped again? That could be a clue as to what it could maybe be.

Well, to give you an example of what I was seeing…

[08-03-15 8:55:54 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 55764 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-03-15 8:55:54 am] | Task (135) machine21-22 has started.
[08-03-15 8:56:04 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:04 am] | Task (136) machine21-22 is new!
[08-03-15 8:56:04 am] | Task (136) /images/image06162015 image file found.
[08-03-15 8:56:04 am] | Task (136) 1 client(s) found.
[08-03-15 8:56:04 am] | Task (136) machine21-22 sending on base port: 53890
[08-03-15 8:56:04 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 53890 --interface bond0 --mcast-data-address 239.111.34.241 --full-duplex --ttl 32 --nokbd;
[08-03-15 8:56:04 am] | Task (136) machine21-22 has started.
[08-03-15 8:56:14 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:14 am] | Task (136) machine21-22 is already running PID 1683
[08-03-15 8:56:24 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:24 am] | Task (136) machine21-22 is already running PID 1683
[08-03-15 8:56:34 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:34 am] | Task (136) machine21-22 is already running PID 1683

So as you see here… On Fog GUI, I tell it to multicast to group called machine21-22, 2 machines in the group. Both machines are configured the same way node wise, so it isn’t separate images. This was what happened before I reverted back the fog/services MulticastTask.class.php that our CFengine messed with from the 0.3.2. Which I thought was related to this issue. When I restarted the service and tried to do the task… I started seeing this in the logs:

[08-03-15 10:22:36 am] * Starting FOG Multicast Manager Service
[08-03-15 10:22:41 am] * Checking for new tasks every 10 seconds.
[08-03-15 10:22:41 am] * Starting service loop.
[08-03-15 10:22:41 am] | Task (138) machine21-22 is new!
[08-03-15 10:22:41 am] | Task (138) /images/image06162015 image file found.
[08-03-15 10:22:41 am] | Task (138) 2 client(s) found.
[08-03-15 10:22:41 am] | Task (138) machine21-22 sending on base port: 53480
[08-03-15 10:22:41 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 2 --portbase 53480 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-03-15 10:22:41 am] | Task (138) machine21-22 has started.
[08-03-15 10:22:51 am] | Task (138) machine21-22 is already running PID 2146
[08-03-15 10:23:01 am] | Task (138) machine21-22 is already running PID 2146
[08-03-15 10:23:11 am] | Task (138) machine21-22 is already running PID 2146
[08-03-15 10:23:21 am] | Task (138) machine21-22 is already running PID 2146

This is the expected behavior I should see. Now besides the know quirk I read that trying to kill this task does not clean up properly (Which I manually killed and not worried about at the moment) The next day, I try to do this test…

[08-04-15 10:53:25 am] * No tasks found!
[08-04-15 10:53:35 am] * No tasks found!
[08-04-15 10:53:45 am] * No tasks found!
[08-04-15 10:53:55 am] | Task (140) machine21-22 is new!
[08-04-15 10:53:55 am] | Task (140) /images/image06162015 image file found.
[08-04-15 10:53:55 am] | Task (140) 1 client(s) found.
[08-04-15 10:53:55 am] | Task (140) machine21-22 sending on base port: 58858
[08-04-15 10:53:55 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 58858 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-04-15 10:53:55 am] | Task (140) machine21-22 has started.
[08-04-15 10:54:05 am] | Task (140) machine21-22 is already running PID 3858
[08-04-15 10:54:05 am] | Task (141) machine21-22 is new!
[08-04-15 10:54:05 am] | Task (141) /images/image06162015 image file found.
[08-04-15 10:54:05 am] | Task (141) 1 client(s) found.
[08-04-15 10:54:05 am] | Task (141) machine21-22 sending on base port: 52754
[08-04-15 10:54:05 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 52754 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-04-15 10:54:05 am] | Task (141) machine21-22 has started.
[08-04-15 10:54:15 am] | Task (140) machine21-22 is already running PID 3858
[08-04-15 10:54:15 am] | Task (141) machine21-22 is already running PID 3886
[08-04-15 10:54:25 am] | Task (140) machine21-22 is already running PID 3858
[08-04-15 10:54:25 am] | Task (141) machine21-22 is already running PID 3886

So what I am seeing is, for some odd reason, the $this->getClientCount() is not getting the right number, but will generate multiple multicast tasks for them individually. Which I don’t understand.

[SKasai]

@Tom-Elliott

@Tom-Elliott said:

You may be willing to try SVN/Trunk/GIT or whatever you want to call it of FOG. Development is what I try to call it, but I do refer often to trunk or svn as well.

It shouldn’t have the problems you’re seeing, and should work fairly well. I’m aware of a quirk or two but it seems to work fine.

Please give it a shot.

There’s also alot of added functionality.

At the moment, this is a production system and somewhat hesitant to go with development code as we also are using this to do a ‘flip-flop’ method with the modified BootMenu… Here is the listing of files that were modded not counting the ones I am trying to tweak…

-rw-r–r-- 1 apache apache 3523 Dec 30 2014 Config.class.php
-rwxr-xr-x 1 root root 27801 Dec 30 2014 BootMenu.class.php.linux
-rwxr-xr-x 1 root root 27801 Dec 30 2014 BootMenu.class.php.windows
-rw-r–r-- 1 root root 8238 Mar 20 15:10 MulticastTask.class.php.work
-rw-r–r-- 1 apache apache 8238 Aug 4 13:25 MulticastTask.class.php
-rwxr-xr-x 1 root root 27801 Aug 5 08:45 BootMenu.class.php

The bootmenu.class.php is copied from the bootMenu.class.php.linux or .windows depending on when we are doing our ‘flipflop’… The only other thing I can see in Config.class.php is it looks like it has settings defined for our server… With the exception of seeing eth0 instead of bond0, as we did tweak the server to use a bonded network connection for the Multicast… And we did specify this on the GUI side.