FOG 1.2.0 Multicast creates multiple single udpcast sessions

[SKasai]

@isaiah658 said:

@SKasai So are you saying that multicasting was actually working recently after you thought you found the problem but now it stopped again? That could be a clue as to what it could maybe be.

Well, to give you an example of what I was seeing…

[08-03-15 8:55:54 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 55764 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-03-15 8:55:54 am] | Task (135) machine21-22 has started.
[08-03-15 8:56:04 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:04 am] | Task (136) machine21-22 is new!
[08-03-15 8:56:04 am] | Task (136) /images/image06162015 image file found.
[08-03-15 8:56:04 am] | Task (136) 1 client(s) found.
[08-03-15 8:56:04 am] | Task (136) machine21-22 sending on base port: 53890
[08-03-15 8:56:04 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 53890 --interface bond0 --mcast-data-address 239.111.34.241 --full-duplex --ttl 32 --nokbd;
[08-03-15 8:56:04 am] | Task (136) machine21-22 has started.
[08-03-15 8:56:14 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:14 am] | Task (136) machine21-22 is already running PID 1683
[08-03-15 8:56:24 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:24 am] | Task (136) machine21-22 is already running PID 1683
[08-03-15 8:56:34 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:34 am] | Task (136) machine21-22 is already running PID 1683

So as you see here… On Fog GUI, I tell it to multicast to group called machine21-22, 2 machines in the group. Both machines are configured the same way node wise, so it isn’t separate images. This was what happened before I reverted back the fog/services MulticastTask.class.php that our CFengine messed with from the 0.3.2. Which I thought was related to this issue. When I restarted the service and tried to do the task… I started seeing this in the logs:

[08-03-15 10:22:36 am] * Starting FOG Multicast Manager Service
[08-03-15 10:22:41 am] * Checking for new tasks every 10 seconds.
[08-03-15 10:22:41 am] * Starting service loop.
[08-03-15 10:22:41 am] | Task (138) machine21-22 is new!
[08-03-15 10:22:41 am] | Task (138) /images/image06162015 image file found.
[08-03-15 10:22:41 am] | Task (138) 2 client(s) found.
[08-03-15 10:22:41 am] | Task (138) machine21-22 sending on base port: 53480
[08-03-15 10:22:41 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 2 --portbase 53480 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-03-15 10:22:41 am] | Task (138) machine21-22 has started.
[08-03-15 10:22:51 am] | Task (138) machine21-22 is already running PID 2146
[08-03-15 10:23:01 am] | Task (138) machine21-22 is already running PID 2146
[08-03-15 10:23:11 am] | Task (138) machine21-22 is already running PID 2146
[08-03-15 10:23:21 am] | Task (138) machine21-22 is already running PID 2146

This is the expected behavior I should see. Now besides the know quirk I read that trying to kill this task does not clean up properly (Which I manually killed and not worried about at the moment) The next day, I try to do this test…

[08-04-15 10:53:25 am] * No tasks found!
[08-04-15 10:53:35 am] * No tasks found!
[08-04-15 10:53:45 am] * No tasks found!
[08-04-15 10:53:55 am] | Task (140) machine21-22 is new!
[08-04-15 10:53:55 am] | Task (140) /images/image06162015 image file found.
[08-04-15 10:53:55 am] | Task (140) 1 client(s) found.
[08-04-15 10:53:55 am] | Task (140) machine21-22 sending on base port: 58858
[08-04-15 10:53:55 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 58858 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-04-15 10:53:55 am] | Task (140) machine21-22 has started.
[08-04-15 10:54:05 am] | Task (140) machine21-22 is already running PID 3858
[08-04-15 10:54:05 am] | Task (141) machine21-22 is new!
[08-04-15 10:54:05 am] | Task (141) /images/image06162015 image file found.
[08-04-15 10:54:05 am] | Task (141) 1 client(s) found.
[08-04-15 10:54:05 am] | Task (141) machine21-22 sending on base port: 52754
[08-04-15 10:54:05 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 52754 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-04-15 10:54:05 am] | Task (141) machine21-22 has started.
[08-04-15 10:54:15 am] | Task (140) machine21-22 is already running PID 3858
[08-04-15 10:54:15 am] | Task (141) machine21-22 is already running PID 3886
[08-04-15 10:54:25 am] | Task (140) machine21-22 is already running PID 3858
[08-04-15 10:54:25 am] | Task (141) machine21-22 is already running PID 3886

So what I am seeing is, for some odd reason, the $this->getClientCount() is not getting the right number, but will generate multiple multicast tasks for them individually. Which I don’t understand.

[SKasai]

@Tom-Elliott

@Tom-Elliott said:

You may be willing to try SVN/Trunk/GIT or whatever you want to call it of FOG. Development is what I try to call it, but I do refer often to trunk or svn as well.

It shouldn’t have the problems you’re seeing, and should work fairly well. I’m aware of a quirk or two but it seems to work fine.

Please give it a shot.

There’s also alot of added functionality.

At the moment, this is a production system and somewhat hesitant to go with development code as we also are using this to do a ‘flip-flop’ method with the modified BootMenu… Here is the listing of files that were modded not counting the ones I am trying to tweak…

-rw-r–r-- 1 apache apache 3523 Dec 30 2014 Config.class.php
-rwxr-xr-x 1 root root 27801 Dec 30 2014 BootMenu.class.php.linux
-rwxr-xr-x 1 root root 27801 Dec 30 2014 BootMenu.class.php.windows
-rw-r–r-- 1 root root 8238 Mar 20 15:10 MulticastTask.class.php.work
-rw-r–r-- 1 apache apache 8238 Aug 4 13:25 MulticastTask.class.php
-rwxr-xr-x 1 root root 27801 Aug 5 08:45 BootMenu.class.php

The bootmenu.class.php is copied from the bootMenu.class.php.linux or .windows depending on when we are doing our ‘flipflop’… The only other thing I can see in Config.class.php is it looks like it has settings defined for our server… With the exception of seeing eth0 instead of bond0, as we did tweak the server to use a bonded network connection for the Multicast… And we did specify this on the GUI side.

[Tom Elliott]

While I understand the hesitation, I don’t know how much help I can provide.

This is especially important to know because of the files you have edited.

I don’t know what the state of the system is. It’s really hard to fix something when others are playing with other things.

As you described, you guys have changed a number of files.

While there are likely some bugs in what we did originally, this is exacerbated greatly by any changes.

One of the quickest fixes, to at least hopefully attempt helping you along.

Attempt clearing out the MulticastSessions and MulticastSessionsAssoc tables;
truncate table multicastSessions;truncate table multicastSessionsAssoc

Then restart the FOGMulticastManager service and create your tasks.

My guess is there are other jobs that the clients are trying to attach themselves to. Truncating should at least fix it for the first time around.

[SKasai]

@Tom-Elliott Thanks for the reply, Tom.

I did try the truncate command as you suggested, but got the following:

truncate table multicastSessions;truncate table multicastSessionsAssoc
truncate: you must specify one of ‘–size’ or ‘–reference’
Try ‘truncate --help’ for more information.
truncate: you must specify one of ‘–size’ or ‘–reference’
Try ‘truncate --help’ for more information.

FYI this is on a Centos 6.6 Machine

I can also send you what we did with those particular files and specify why given our situation in a private chat.

[SKasai]

My apologies… Apparently, I missed the fact I need to do this in the mysql db… The results were:

mysql> truncate table multicastSessions; truncate table multicastSessionsAssoc;
Query OK, 0 rows affected (0.04 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> exit

However, testing it, it shows the following:

[08-05-15 11:21:18 am] * Starting FOG Multicast Manager Service
[08-05-15 11:21:23 am] * Checking for new tasks every 10 seconds.
[08-05-15 11:21:23 am] * Starting service loop.
[08-05-15 11:21:23 am] * No tasks found!
[08-05-15 11:21:33 am] * No tasks found!
[08-05-15 11:21:43 am] * No tasks found!
[08-05-15 11:21:53 am] * No tasks found!
[08-05-15 11:22:03 am] * No tasks found!
[08-05-15 11:22:13 am] * No tasks found!
[08-05-15 11:22:23 am] | Task (1) machine21-22 is new!
[08-05-15 11:22:23 am] | Task (1) /images/image06162015 image file found.
[08-05-15 11:22:23 am] | Task (1) 1 client(s) found.
[08-05-15 11:22:23 am] | Task (1) machine21-22 sending on base port: 64180
[08-05-15 11:22:23 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 64180 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-05-15 11:22:23 am] | Task (1) machine21-22 has started.
[08-05-15 11:22:34 am] | Task (1) machine21-22 is already running PID 18543
[08-05-15 11:22:34 am] | Task (2) machine21-22 is new!
[08-05-15 11:22:34 am] | Task (2) /images/image06162015 image file found.
[08-05-15 11:22:34 am] | Task (2) 1 client(s) found.
[08-05-15 11:22:34 am] | Task (2) machine21-22 sending on base port: 54332
[08-05-15 11:22:34 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 54332 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-05-15 11:22:34 am] | Task (2) machine21-22 has started.
[08-05-15 11:22:44 am] | Task (1) machine21-22 is already running PID 18543
[08-05-15 11:22:44 am] | Task (2) machine21-22 is already running PID 18563
[08-05-15 11:22:54 am] | Task (1) machine21-22 is already running PID 18543
[08-05-15 11:22:54 am] | Task (2) machine21-22 is already running PID 18563

[Tom Elliott]

@SKasai Those are SQL commands. Not regular bash commands.

You need to login to the mysql server and run those truncate commands.

[SKasai]

@Tom-Elliott Yup, realized the mistake and posted a follow up… Unfortunately, yielded the same results. See follow up message.

I am stumped as to what is passing over to the Fogmulticastmanager to do it this way.

[Wayne Workman]

@SKasai said:

@Tom-Elliott Yup, realized the mistake and posted a follow up… Unfortunately, yielded the same results. See follow up message.

I am stumped as to what is passing over to the Fogmulticastmanager to do it this way.

Just going to remind you that there have been improvements made to multicast in FOG Trunk.

You’ve had this problem for 5 months now??? Would it really be too out-of-the-way to spend a day setting up a test environment and trying FOG Trunk?

It’s simple… I’ll explain.

• Don’t touch your production server. Setup a new fog trunk server and install it with DHCP enabled (you may use an old desktop for this).

• Pick a computer lab - take the image that lab needs and copy it to your new fog server.

• Export hosts from production server and import to new server.

• Move the new fog server (physically) to that computer lab’s switch. Unplug the switch’s uplink. Connect the fog server to that switch.

• Try to multicast, observe, report back.

[Wayne Workman]

Also, other simple questions I thought of…

• Is the computers in your multicast group all connected to the same switch? Are you sure? Did you actually go look to see?

• Are all of the patch cables good? One single patch cable operating at 100mbps could throw everything off.

• Are the ports on the switch that the multicast group is connected to all on the same VLAN? If not, it would probably create the problem you are seeing.

• Are all the computers in the group on the same subnet?

[SKasai]

@Wayne-Workman Tom just chatted with me on a side channel and I will be working on another machine with the new rev with some implementations. The reason I was hesitant also is that by going to the new rev what changes will need to be ‘hacked’ again. So Tom showed me some of the new rev’s stuff, part of which includes the boot menu which our way of doing it will have to stop and be redone but Tom gave me some pointers and a script for what our situation is and once I have it working correctly, I will post more on it.

To address some of your questions, my main focus was on the fact our server was creating single client multicast jobs for each node versus 1 multicast job for x clients. We did a few things and it didn’t resolve the issue on that part.

He states it may be related to the 1.2.0’s hosts.class.php having a bug and should be fixed in the latest trunk build, which I will have to do a test build on another machine first. So the problem is less about connecting to the same switch and what not but seeing the problem being at the server to begin with first. The hardware/vlan issue I can beat on my networking guy over later with for the connectivity because that will then be our ACL/routing issues which I can tackle at another time.

As for getting back to this 5 months later, partly this is due to being mostly busy and our next reimage of the lab will be in a month or two away and things have finally slowed down to look at this again.

[SKasai]

So worked with Tom and being on Trunk 3475 and up, the multicast task creation issue has been resolved. Testing the new trunk version has been fairly consistent in creating single multicast tasks versus multiple multicast tasks with single clients.