Ok… Then basically for now, just have a hack shellscript to do…
#!/bin/sh
/usr/bin/systemctl restart FOGImageReplicator.service
/usr/bin/systemctl restart FOGMulticastManager.service
/usr/bin/systemctl restart FOGScheduler.service
/usr/bin/systemctl restart FOGSnapinReplicator.service
Got it. 
(Yes, I know it can be manually started, but easier just to do this for ‘deadhead’ purposes).
May have found a possible bug with CentOS 7, with MariaDB and FOG build 4375…
Rebooting CentOS 7, checking Systemctl, seeing the following:
FOGImageReplicator.service loaded inactive dead FOGImageReplicator
FOGMulticastManager.service loaded inactive dead FOGMulticastManager
FOGScheduler.service loaded inactive dead FOGScheduler
FOGSnapinReplicator.service loaded inactive dead FOGSnapinReplicator
Doing a systemctl status on one of the services…
FOGScheduler.service - FOGScheduler
Loaded: loaded (/usr/lib/systemd/system/FOGScheduler.service; enabled)
Active: inactive (dead) since Wed 2015-08-12 13:09:50 PDT; 3min 32s ago
Process: 878 ExecStart=/opt/fog/service/FOGTaskScheduler/FOGTaskScheduler & (code=exited, status=0/SUCCESS)
Main PID: 878 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/FOGScheduler.service
Aug 12 13:09:43 control systemd[1]: Started FOGScheduler.
Aug 12 13:09:50 control FOGTaskScheduler[878]: PHP Warning: mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/db/MySQL.class.php on line 37
Aug 12 13:09:50 control FOGTaskScheduler[878]: PHP Warning: mysqli::mysqli(): (HY000/2002): Connection refused in /var/www/html/fog/lib/db/MySQL.class.php on line 39
Aug 12 13:09:50 control FOGTaskScheduler[878]: PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/html/fog/lib/db/MySQL.class.php on line 63
However, when you try and restart it, works fine. So this could be a timing issue with MariaDB and/or CentOS 7…
So worked with Tom and being on Trunk 3475 and up, the multicast task creation issue has been resolved. Testing the new trunk version has been fairly consistent in creating single multicast tasks versus multiple multicast tasks with single clients.
One more thing to clarify as well… I have also tried replacing the ${boot-url} with the ip of the fog server, thinking maybe that had something to do with it, but not the case here… I also checked to see that service/ipxe/grub.exe does exist (Which it should, since I think this is also the same format used by the default bootatexit within the BootMenu.class.php file as well.
So trying SVN Build 4375 and ran into a small problem with the iPXE boot Menu option where we have a lab of computers with 2 hard drives. hd0 which has Linux and hd1 with Windows.
I have one machine set to go to our SVN4375 fog machine to make the boot menu a little more ‘cleaner’ with Tom’s help, however, seeing the following reaction:
If left alone, Boot from Hard Disk Option works to boot hd0.
Trying to use the defined fog.local.linux or fog.local.windows with the defined parameter:
chain -ar http://${boot-url}/service/ipxe/grub.exe --config-file=“rootnoverify (hd1);chainloader +1” || goto MENU
This loops back to the menu under the state the menu options are (Tom gave me a script to define a ‘use this as default’ for time jobs).
Manually selecting the alternate menu items yields the same problem, so it never really goes to ‘boot’ the option we would like. The Boot Menu has the exit type set to ‘grub style’. I have also tried doing ‘exit style’ and ‘sanboot style’ as well and still running into the same problem with the custom boot drive menu options…
The only real difference between linux and windows on the quoted parameter is hd0 for linux and hd1 for windows.
Is there something else I am missing somewhere?
@Wayne-Workman Tom just chatted with me on a side channel and I will be working on another machine with the new rev with some implementations. The reason I was hesitant also is that by going to the new rev what changes will need to be ‘hacked’ again. So Tom showed me some of the new rev’s stuff, part of which includes the boot menu which our way of doing it will have to stop and be redone but Tom gave me some pointers and a script for what our situation is and once I have it working correctly, I will post more on it.
To address some of your questions, my main focus was on the fact our server was creating single client multicast jobs for each node versus 1 multicast job for x clients. We did a few things and it didn’t resolve the issue on that part.
He states it may be related to the 1.2.0’s hosts.class.php having a bug and should be fixed in the latest trunk build, which I will have to do a test build on another machine first. So the problem is less about connecting to the same switch and what not but seeing the problem being at the server to begin with first. The hardware/vlan issue I can beat on my networking guy over later with for the connectivity because that will then be our ACL/routing issues which I can tackle at another time.
As for getting back to this 5 months later, partly this is due to being mostly busy and our next reimage of the lab will be in a month or two away and things have finally slowed down to look at this again.
@Tom-Elliott Yup, realized the mistake and posted a follow up… Unfortunately, yielded the same results. See follow up message.
I am stumped as to what is passing over to the Fogmulticastmanager to do it this way.
My apologies… Apparently, I missed the fact I need to do this in the mysql db… The results were:
mysql> truncate table multicastSessions; truncate table multicastSessionsAssoc;
Query OK, 0 rows affected (0.04 sec)
Query OK, 0 rows affected (0.00 sec)
mysql> exit
However, testing it, it shows the following:
[08-05-15 11:21:18 am] * Starting FOG Multicast Manager Service
[08-05-15 11:21:23 am] * Checking for new tasks every 10 seconds.
[08-05-15 11:21:23 am] * Starting service loop.
[08-05-15 11:21:23 am] * No tasks found!
[08-05-15 11:21:33 am] * No tasks found!
[08-05-15 11:21:43 am] * No tasks found!
[08-05-15 11:21:53 am] * No tasks found!
[08-05-15 11:22:03 am] * No tasks found!
[08-05-15 11:22:13 am] * No tasks found!
[08-05-15 11:22:23 am] | Task (1) machine21-22 is new!
[08-05-15 11:22:23 am] | Task (1) /images/image06162015 image file found.
[08-05-15 11:22:23 am] | Task (1) 1 client(s) found.
[08-05-15 11:22:23 am] | Task (1) machine21-22 sending on base port: 64180
[08-05-15 11:22:23 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 64180 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-05-15 11:22:23 am] | Task (1) machine21-22 has started.
[08-05-15 11:22:34 am] | Task (1) machine21-22 is already running PID 18543
[08-05-15 11:22:34 am] | Task (2) machine21-22 is new!
[08-05-15 11:22:34 am] | Task (2) /images/image06162015 image file found.
[08-05-15 11:22:34 am] | Task (2) 1 client(s) found.
[08-05-15 11:22:34 am] | Task (2) machine21-22 sending on base port: 54332
[08-05-15 11:22:34 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 54332 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-05-15 11:22:34 am] | Task (2) machine21-22 has started.
[08-05-15 11:22:44 am] | Task (1) machine21-22 is already running PID 18543
[08-05-15 11:22:44 am] | Task (2) machine21-22 is already running PID 18563
[08-05-15 11:22:54 am] | Task (1) machine21-22 is already running PID 18543
[08-05-15 11:22:54 am] | Task (2) machine21-22 is already running PID 18563
@Tom-Elliott Thanks for the reply, Tom.
I did try the truncate command as you suggested, but got the following:
truncate table multicastSessions;truncate table multicastSessionsAssoc
truncate: you must specify one of ‘–size’ or ‘–reference’
Try ‘truncate --help’ for more information.
truncate: you must specify one of ‘–size’ or ‘–reference’
Try ‘truncate --help’ for more information.
FYI this is on a Centos 6.6 Machine
I can also send you what we did with those particular files and specify why given our situation in a private chat.
@Tom-Elliott said:
You may be willing to try SVN/Trunk/GIT or whatever you want to call it of FOG. Development is what I try to call it, but I do refer often to trunk or svn as well.
It shouldn’t have the problems you’re seeing, and should work fairly well. I’m aware of a quirk or two but it seems to work fine.
Please give it a shot.
There’s also alot of added functionality.
At the moment, this is a production system and somewhat hesitant to go with development code as we also are using this to do a ‘flip-flop’ method with the modified BootMenu… Here is the listing of files that were modded not counting the ones I am trying to tweak…
-rw-r–r-- 1 apache apache 3523 Dec 30 2014 Config.class.php
-rwxr-xr-x 1 root root 27801 Dec 30 2014 BootMenu.class.php.linux
-rwxr-xr-x 1 root root 27801 Dec 30 2014 BootMenu.class.php.windows
-rw-r–r-- 1 root root 8238 Mar 20 15:10 MulticastTask.class.php.work
-rw-r–r-- 1 apache apache 8238 Aug 4 13:25 MulticastTask.class.php
-rwxr-xr-x 1 root root 27801 Aug 5 08:45 BootMenu.class.php
The bootmenu.class.php is copied from the bootMenu.class.php.linux or .windows depending on when we are doing our ‘flipflop’… The only other thing I can see in Config.class.php is it looks like it has settings defined for our server… With the exception of seeing eth0 instead of bond0, as we did tweak the server to use a bonded network connection for the Multicast… And we did specify this on the GUI side.
@isaiah658 said:
@SKasai So are you saying that multicasting was actually working recently after you thought you found the problem but now it stopped again? That could be a clue as to what it could maybe be.
Well, to give you an example of what I was seeing…
[08-03-15 8:55:54 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 55764 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-03-15 8:55:54 am] | Task (135) machine21-22 has started.
[08-03-15 8:56:04 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:04 am] | Task (136) machine21-22 is new!
[08-03-15 8:56:04 am] | Task (136) /images/image06162015 image file found.
[08-03-15 8:56:04 am] | Task (136) 1 client(s) found.
[08-03-15 8:56:04 am] | Task (136) machine21-22 sending on base port: 53890
[08-03-15 8:56:04 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 53890 --interface bond0 --mcast-data-address 239.111.34.241 --full-duplex --ttl 32 --nokbd;
[08-03-15 8:56:04 am] | Task (136) machine21-22 has started.
[08-03-15 8:56:14 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:14 am] | Task (136) machine21-22 is already running PID 1683
[08-03-15 8:56:24 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:24 am] | Task (136) machine21-22 is already running PID 1683
[08-03-15 8:56:34 am] | Task (135) machine21-22 is already running PID 1672
[08-03-15 8:56:34 am] | Task (136) machine21-22 is already running PID 1683
So as you see here… On Fog GUI, I tell it to multicast to group called machine21-22, 2 machines in the group. Both machines are configured the same way node wise, so it isn’t separate images. This was what happened before I reverted back the fog/services MulticastTask.class.php that our CFengine messed with from the 0.3.2. Which I thought was related to this issue. When I restarted the service and tried to do the task… I started seeing this in the logs:
[08-03-15 10:22:36 am] * Starting FOG Multicast Manager Service
[08-03-15 10:22:41 am] * Checking for new tasks every 10 seconds.
[08-03-15 10:22:41 am] * Starting service loop.
[08-03-15 10:22:41 am] | Task (138) machine21-22 is new!
[08-03-15 10:22:41 am] | Task (138) /images/image06162015 image file found.
[08-03-15 10:22:41 am] | Task (138) 2 client(s) found.
[08-03-15 10:22:41 am] | Task (138) machine21-22 sending on base port: 53480
[08-03-15 10:22:41 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 2 --portbase 53480 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-03-15 10:22:41 am] | Task (138) machine21-22 has started.
[08-03-15 10:22:51 am] | Task (138) machine21-22 is already running PID 2146
[08-03-15 10:23:01 am] | Task (138) machine21-22 is already running PID 2146
[08-03-15 10:23:11 am] | Task (138) machine21-22 is already running PID 2146
[08-03-15 10:23:21 am] | Task (138) machine21-22 is already running PID 2146
This is the expected behavior I should see. Now besides the know quirk I read that trying to kill this task does not clean up properly (Which I manually killed and not worried about at the moment) The next day, I try to do this test…
[08-04-15 10:53:25 am] * No tasks found!
[08-04-15 10:53:35 am] * No tasks found!
[08-04-15 10:53:45 am] * No tasks found!
[08-04-15 10:53:55 am] | Task (140) machine21-22 is new!
[08-04-15 10:53:55 am] | Task (140) /images/image06162015 image file found.
[08-04-15 10:53:55 am] | Task (140) 1 client(s) found.
[08-04-15 10:53:55 am] | Task (140) machine21-22 sending on base port: 58858
[08-04-15 10:53:55 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 58858 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-04-15 10:53:55 am] | Task (140) machine21-22 has started.
[08-04-15 10:54:05 am] | Task (140) machine21-22 is already running PID 3858
[08-04-15 10:54:05 am] | Task (141) machine21-22 is new!
[08-04-15 10:54:05 am] | Task (141) /images/image06162015 image file found.
[08-04-15 10:54:05 am] | Task (141) 1 client(s) found.
[08-04-15 10:54:05 am] | Task (141) machine21-22 sending on base port: 52754
[08-04-15 10:54:05 am] CMD: cat /images/image06162015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 52754 --interface bond0 --mcast-data-address 239.x.x.x --full-duplex --ttl 32 --nokbd;
[08-04-15 10:54:05 am] | Task (141) machine21-22 has started.
[08-04-15 10:54:15 am] | Task (140) machine21-22 is already running PID 3858
[08-04-15 10:54:15 am] | Task (141) machine21-22 is already running PID 3886
[08-04-15 10:54:25 am] | Task (140) machine21-22 is already running PID 3858
[08-04-15 10:54:25 am] | Task (141) machine21-22 is already running PID 3886
So what I am seeing is, for some odd reason, the $this->getClientCount() is not getting the right number, but will generate multiple multicast tasks for them individually. Which I don’t understand.
Well, looks like I may have jumped the gun on that conclusion… A day later, the problem is back and still trying to figure out the issue…
But to put this in a little better context for all of you… Here is the hack on the file for /var/www/html/fog/lib/fog/MulticastTask.class.php that we were using… The First block is our hack, the Second block the Original
92,93c92,93
$cmd = 'cat '.$strRec.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;'; $cmd .= 'cat '.$strSys.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';
$cmd = 'cat '.$strRec.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;'; $cmd .= 'cat '.$strSys.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';
96c96
$cmd = 'cat '.$strSys.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';
$cmd = 'cat '.$strSys.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';
121c121
$cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';
$cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';
125c125
$cmd = 'cat '.rtrim($this->getImagePath(),'/').'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';
$cmd = 'cat '.rtrim($this->getImagePath(),'/').'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';
134c134
if ($handle = opendir($this->getImagePath()))
if($handle = opendir($this->getImagePath()))
153c153
$cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';
$cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';
178c178
$cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' --mcast-data-address 239.x.x.x '.$wait.' --full-duplex --ttl 32 --nokbd;';
$cmd .= 'cat '.$path.'|'.UDPSENDERPATH.' --min-receivers '.$this->getClientCount().' --portbase '.$this->getPortBase().' '.$interface.' '.$wait.' --full-duplex --ttl 32 --nokbd;';
So basically, we just added --mcast-data-address 239.x.x.x to force it to do the broadcast to the 239.x.x.x for the Cisco to allow it to work properly… What we are seeing, though is that when it creates command, it seems to do $this->getClientCount() as ‘1’ instead of the number of clients, but generates the clientcount’s # of processes.
Sorry for taking so long to get back to everyone here… A lot of busy work and had to put this on the back burner…
Found the issue that was causing the problem. We are currently using cfengine and it pushed the 0.3.2 version of MulticastTask.class.php, which is completely not the right one for the 1.2.0, for obvious reasons, but was able to generate the right command with the wrong reasons.
Restoring the original MulticastTask.class.php that comes with 1.2.0 seems to have fixed it.
The reason we had cfengine pushing a modified version of the 0.3.2 config was due to needing to ‘hack’ the multicasttask.class.php to make it work a little easier with the Cisco switches. This was done prior to my time so I don’t know what it was exactly.
<?php
define( “WEBROOT”, “/var/www/html/fog” );
?>
Well, looking at /opt/fog/service/common/lib…
We had to hack the MulticastTask.class.php for some reason… This is what we have:
[url=“/_imported_xf_attachments/1/1833_MulticastTask.class.php?:”]MulticastTask.class.php[/url]
As mentioned above, running CentOS 6.6, using FOG version 1.2.0.
Multicast did work earlier, however, I believe, at one point we had to tweak something with regards to ipxe boot.php as we have one lab where it needs to ‘flip-flop’ between booting from one drive to another (sda - Linux, sdb - Windows), which the default fog ipxe did not allow at the time…
Group -> Search for Group -> Select Multicast -> Create Multicast Job.
This would be for FOG 1.2.0, running on CentOS 6.6
Yes. We even tried doing a simple 2 computer test and the resultant was seeing both machines starting their own separate UDP cast session. Image source is the same. Example from the multicast log:
[03-20-15 4:50:45 pm] | Task (131) xxx56-57 is new!
[03-20-15 4:50:45 pm] | Task (131) /images/03132015 image file found.
[03-20-15 4:50:45 pm] | Task (131) 1 client(s) found.
[03-20-15 4:50:45 pm] | Task (131) xxx56-57 sending on base port: 51130
[03-20-15 4:50:45 pm] CMD: cat /images/03132015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 51130 --interface bond0 --mcast-data-address 239.xxx.xxx.241 --full-duplex --ttl 32 --nokbd;
[03-20-15 4:50:45 pm] | Task (131) xxx56-57 has started.
[03-20-15 4:50:56 pm] | Task (131) xxx56-57 is already running PID 32173
[03-20-15 4:50:56 pm] | Task (132) xxx56-57 is new!
[03-20-15 4:50:56 pm] | Task (132) /images/03132015 image file found.
[03-20-15 4:50:56 pm] | Task (132) 1 client(s) found.
[03-20-15 4:50:56 pm] | Task (132) xxx56-57 sending on base port: 60678
[03-20-15 4:50:56 pm] CMD: cat /images/03132015/d1p1.img|/usr/local/sbin/udp-sender --min-receivers 1 --portbase 60678 --interface bond0 --mcast-data-address 239.xxx.xxx.241 --full-duplex --ttl 32 --nokbd;
[03-20-15 4:50:56 pm] | Task (132) xxx56-57 has started.
As you see here… This is one job, yet it generates two UDP sessions for 2 machines. This is not me spamming the task in succession. You will also notice it says ‘–min-receivers 1’… It should say ‘2’
I understand this. The problem I am seeing is the following:
We define a group of 10 machines to do a Multicast image.
On the FOG server, command line, I see instead of 2 UDP based processes, we have 20 of them, 1 to send, one to receive.
Multicast.log shows it is creating 10 seperate UDP Multicast jobs with only 1 member, instead of 1 UDP multicast job waiting for 10 members.
My issue is trying to find out what, in a group where we specify a multicast job, it creates 10 separate UDP casts for 1 member jobs instead of 1 UDP cast waiting for 10 members job.