Fog 1.1.0 multicast sits at "Starting to restore image (-) to device (/dev/sda1)

t.mayer

I have the same multicast-problem with Version 1.1.2 of fog.
OS: Debian 7.5 x64 on ESXI 5.5
Before that everything was working perfekt with Version 0.32 of FOG and x86-Linux.
Is it possible that it has to do something with the 32-to-64-Bit change of the OS?

ianabc

OK, in my case the problem is fixed and it was my network (sigh…:). The short story is, I should have started testing at the most basic level by trying to transfer anything by multicast, I’ve found omping and udpcast useful in doing this. If someone has a good understanding of multicast I think some debugging examples for using udpcast or omping would be a nice addition to the wiki.

Now for the long answer…

My test setup uses a KVM/QMEU network along with 3 KVM guests: One fog server and two client machines. All the guests are networked together on 192.168.222.0/24 which is NATed to give access to the outside world. The NATing is done by iptables on the KVM host via libvirt and has a PREROUTING chain which looks like
[CODE]
$ iptables -t nat -nL
…
RETURN all – 192.168.222.0/24 224.0.0.0/24
RETURN all – 192.168.222.0/24 255.255.255.255
MASQUERADE tcp – 192.168.222.0/24 !192.168.222.0/24 masq ports: 1024-65535
MASQUERADE udp – 192.168.222.0/24 !192.168.222.0/24 masq ports: 1024-65535

MASQUERADE  all  --  192.168.222.0/24    !192.168.222.0/24

[/CODE]
The problem is the first of those lines, which applies only to 224.0.0.0/24, when I want it to include all mutlciast addresses 224.0.0.0/4. Making this change allows omping, udpcast and [I]of course[/I] fog to use multicast without the hanging problem!

BullDozer

I just did another clean install of Ubuntu Desktop 13.10, with FOG 1.1.2. I made sure that I did not set a MYSQL password.
I still cannot multicast.

[07-16-14 4:59:22 pm] * No tasks found!
[07-16-14 4:59:32 pm] | Task (1) Multi is new!
[07-16-14 4:59:32 pm] | Task (1) /images/163QAIfinalRev5 image file found.
[07-16-14 4:59:32 pm] | Task (1) 3 client(s) found.
[07-16-14 4:59:32 pm] | Task (1) Multi sending on base port: 63100
[07-16-14 4:59:32 pm] CMD: cat “/images/163QAIfinalRev5”|/usr/local/sbin/udp-sender --min-receivers 3 --portbase 63100 --interface eth0 --full-duplex --ttl 32 --nokbd;
[07-16-14 4:59:32 pm] | Task (1) Multi has started.
[07-16-14 4:59:42 pm] | Task (1) Multi is no longer running.
[07-16-14 4:59:52 pm] | Task (1) Multi is no longer running.

I have deleted the items from the table and restarted the services and nada.
I can single download and upload without issue.

ianabc

I found it useful to start debugging with omping and udpcast directly before trying to debug fog - it means you get instant answers, you don’t have to wait for a client to reboot.

If you can install omping somewhere (like a spare linux client), then you can try
[CODE]
fog-server$ omping FOG.CLIENT.IP FOG.SERVER.IP
…

fog-client$ omping FOG.SERVER.IP FOG.CLIENT.IP
[/CODE]
On my system I see output on the fog server similar to
[CODE]
192.168.222.102 : multicast, seq=37, size=69 bytes, dist=0, time=0.437ms
192.168.222.102 : unicast, seq=38, size=69 bytes, dist=0, time=0.482ms
192.168.222.102 : multicast, seq=38, size=69 bytes, dist=0, time=0.495ms
192.168.222.102 : unicast, seq=39, size=69 bytes, dist=0, time=0.485ms
192.168.222.102 : multicast, seq=39, size=69 bytes, dist=0, time=0.499ms
192.168.222.102 : unicast, seq=40, size=69 bytes, dist=0, time=0.274ms
192.168.222.102 : multicast, seq=40, size=69 bytes, dist=0, time=0.291ms
192.168.222.102 : unicast, seq=41, size=69 bytes, dist=0, time=0.446ms

192.168.222.102 : multicast, seq=41, size=69 bytes, dist=0, time=0.455ms
[/CODE]
When multicast is working. When it isn’t working I only see the unicast responses.

ianabc

Also, I just noticed that I can test multicast with the plain vanilla ping command as long as icmp_echo_ignore_broadcasts is set to 0, e.g.
[CODE]
fog-server$ echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
fog-client$ echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

fog-server$ ping -c 5 224.0.0.1
PING 224.0.0.1 (224.0.0.1) 56(84) bytes of data.
64 bytes from 192.168.222.100: icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from 192.168.222.102: icmp_seq=1 ttl=64 time=0.393 ms (DUP!)
64 bytes from 192.168.222.100: icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from 192.168.222.102: icmp_seq=2 ttl=64 time=0.414 ms (DUP!)
64 bytes from 192.168.222.100: icmp_seq=3 ttl=64 time=0.041 ms
64 bytes from 192.168.222.102: icmp_seq=3 ttl=64 time=0.402 ms (DUP!)
64 bytes from 192.168.222.100: icmp_seq=4 ttl=64 time=0.042 ms
64 bytes from 192.168.222.102: icmp_seq=4 ttl=64 time=0.424 ms (DUP!)
64 bytes from 192.168.222.100: icmp_seq=5 ttl=64 time=0.036 ms

--- 224.0.0.1 ping statistics ---
5 packets transmitted, 5 received, +4 duplicates, 0% packet loss, time 3999ms

rtt min/avg/max/mdev = 0.036/0.204/0.424/0.182 ms

[/CODE]
On a network that doesn’t permit multicast, the same thing gives me
[CODE]

fog-server$ ping -c 5 224.0.0.1
PING 224.0.0.1 (224.0.0.1) 56(84) bytes of data.
64 bytes from 192.168.222.100: icmp_seq=1 ttl=64 time=0.046 ms
64 bytes from 192.168.222.100: icmp_seq=2 ttl=64 time=0.041 ms
64 bytes from 192.168.222.100: icmp_seq=3 ttl=64 time=0.044 ms
64 bytes from 192.168.222.100: icmp_seq=4 ttl=64 time=0.040 ms
64 bytes from 192.168.222.100: icmp_seq=5 ttl=64 time=0.042 ms

--- 224.0.0.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.040/0.042/0.046/0.007 ms

[/CODE]
i.e. Only the machine sending the pings responds, the client never sees them.

[B]N.B. I should also point out that I have basically no idea what I’m doing when it comes to multicast so YMMV :-)[/B]

t.mayer

I tried with fog 1.1.2 and debian 7.6 as well as with ubuntu 12.04.

Uploading works.
Downloading works with a single win-8-client (it hangs after the clone-process, but thats another story i guess).
Multicast hangs at “Starting to restore image (-) to device (/dev/sda1)”.

On the same hardware with fog 0.32 and ubuntu 12.04 multicast works like a charm.

Any ideas?
Greeds!

ianabc

t.mayer: Could you try the omping commands above just to confirm that multicast routing is working for you.

t.mayer

[quote=“ianabc, post: 33453, member: 24548”]t.mayer: Could you try the omping commands above just to confirm that multicast routing is working for you.[/quote]
Thanks for the answer ianabc!

I have to be honest: I dont understand the omping-thing…
Isn’t it evidence enough, that it works with fog 0.32?
Can you explaing me in wich state client and server have to be to do the omping?

ianabc

I guess so, as long as you can be absolutely sure there were no network config changes between the 0.32 and 1.1.2 tests.

I don’t really understand multicast myself, but if you can find (or compile) omping somewhere I found it really handy for isolating the problem to multicast. Fog has so many parts that it can take a while to debug, omping lets you confirm that IP multicast routing is working on your network so you have the confidence that the problem is actually with fog.

If this is a development fog server you might want to try truncating the SQL tables that tom mentioned above, and also upgrading to SVN to see if the problem is still there.

Sage

[B][COLOR=#ff0000]EDIT: Sorted! The new SVN helped - it works now like a charm! Thank you![/COLOR][/B]

Good afternoon fellow Foggers!
I have the same problem as stated before - multicast just refuses to work and hangs on the window.

Unicast works just fine, but multicast just refuses to work.

Server: Debian 7.6
Fog: 1.1.2 latest version.

Here is what I get out of my udp-log file.

[CODE]Udp-sender 20120424
Using mcast address 234.10.48.7
UDP sender for (stdin) at 10.10.48.7 on eth0
Broadcasting control to 224.0.0.1
New connection from 10.10.50.70 (#0) 00000009
New connection from 10.10.50.71 (#1) 00000009
New connection from 10.10.50.72 (#2) 00000009
New connection from 10.10.50.73 (#3) 00000009
New connection from 10.10.50.74 (#4) 00000009
New connection from 10.10.50.75 (#5) 00000009
New connection from 10.10.50.77 (#6) 00000009
New connection from 10.10.50.121 (#7) 00000009
New connection from 10.10.50.76 (#8) 00000009
Starting transfer: 00000009
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Timeout notAnswered=[0,1,2,3,4,5,6,7,8] notReady=[0,1,2,3,4,5,6,7,8] nrAns=0 nrRead=0 nrPart=9 avg=10000
Dropping client #0 because of timeout
Disconnecting #0 (10.10.50.70)
Dropping client #1 because of timeout
Disconnecting #1 (10.10.50.71)
Dropping client #2 because of timeout
Disconnecting #2 (10.10.50.72)
Dropping client #3 because of timeout
Disconnecting #3 (10.10.50.73)
Dropping client #4 because of timeout
Disconnecting #4 (10.10.50.74)
Dropping client #5 because of timeout
Disconnecting #5 (10.10.50.75)
Dropping client #6 because of timeout
Disconnecting #6 (10.10.50.77)
Dropping client #7 because of timeout
Disconnecting #7 (10.10.50.121)
Dropping client #8 because of timeout
Disconnecting #8 (10.10.50.76)
bytes= re-xmits=0000000 ( 0.0%) slice=0112 - 0
Transfer complete.[/CODE]

Here are two images from my router. When I launch multicast - the fog control management says that they are in q[SIZE=3][FONT=arial][COLOR=#545454][B]ueue[/B][/COLOR][/FONT][/SIZE] - once I start the clients, the management shows that it’s in “progress”. Here’s what the routers say about it:
Seems like both, the server & the clients are sending signals to each other.

Server is on 10.10.48.7 ip address and the clients are on the 10.10.50. subnet

[IMG]http://www.pixentral.com/pics/1y1KRscWeCz5zGoDid0W0VX8YihKLk.png[/IMG]

[IMG]http://www.pixentral.com/pics/19BdkPufSWuXOIvkk7IHRFuAUJHHPg0.png[/IMG]

Is there any possible solution for this?

TRSD

We are having the same problem on v.1.1.2. We tried omping and that works.

gwhitfield

Pardon if this thread is going out of date but I thought I’d chime in a bit since I’ve been struggling with this on a new install of FOG 1.2.0 on Ubuntu 12.04 and I “think” I’ve got my issue sorted a bit, at least enough that I had success multicasting this morning when it wasn’t working Friday afternoon. Note I’ve also had to figure out the tftpd-hpa restart thing which I don’t get at all. I’m an Ubuntu/Linux noob but I’ve got several FOG 0.29 boxes at different schools on Ubuntu 10.04 that have been running without problems out-of-the-box (so to speak) for YEARS so it’s unfortunate to see this kind of issue in an updated OS but it is what it is. Anyway, my clients were multicasting fine Friday morning on my new install until I shut it down to move it out of the lab and into our server room and a new switch too. All of a sudden I can’t multicast anymore but tasks have been going through this switch just fine all day so that ain’t it. Tons of Googling and forums research later and I get to this thread and a couple other like it so I vpn in from home (when I shouldn’t be working) and …

… from CLI I tried to restart the FOGMulticastManager and got this:

[EMAIL]administrator@GCC-FOGSERVER:~$[/EMAIL] sudo /etc/init.d/FOGMulticastManager restart

Restarting FOG Computer Imaging Solution: FOGMulticastManager
start-stop-daemon: warning: failed to kill 1069: No such process [ OK ]

… so thinking “AHA!! The service wasn’t even running” I tried several times to start the service and got this:

[EMAIL]administrator@GCC-FOGSERVER:~$[/EMAIL] sudo /etc/init.d/FOGMulticastManager start

Starting FOG Computer Imaging Solution: FOGMulticastManager [fail]

… after thinking on it a bit I try to stop and start as individual steps:

[EMAIL]administrator@GCC-FOGSERVER:~$[/EMAIL] sudo /etc/init.d/FOGMulticastManager stop

Stopping FOG Computer Imaging Solution: FOGMulticastManager [ OK ]
[EMAIL]administrator@GCC-FOGSERVER:~$[/EMAIL] sudo /etc/init.d/FOGMulticastManager start
Starting FOG Computer Imaging Solution: FOGMulticastManager [ OK ]

… Oooooooh, that looks good so I try another restart to see what happens:

[EMAIL]administrator@GCC-FOGSERVER:~$[/EMAIL] sudo /etc/init.d/FOGMulticastManager restart

Restarting FOG Computer Imaging Solution: FOGMulticastManager [ OK ]

I was then able to successfully initiate a multicast task and the machines thankfully all woke up and did their thing!! BTW - I LOVE when I can see the progress of a multicast task in 1.2.0 which is the only way I could tell if anything worked since I was at home. It’s my favorite.

Something is goofy on a reboot where (in addition to tftpd-hpa issue) the FOGMulticastManager process can’t be started or restarted because “no such process”. I hope this may help someone who has the problem.

Tom Elliott

There’s a problem, similar to the tftpd-hpa issue, with mysql. I believe this is what you’re encountering.

It’s trying to start mysql, tftpd-hpa, and the FOG Services before the network device is available to run. What ends up happening is the processes (FOGMulticastManager,FOGImageReplicator,FOGScheduler) are running, but can’t communicate. So a restart doesn’t work because the restart simply kills and starts a new process. The previous process can’t stop so it just starts another process using the original process generated as if it were a child process. Because that doesn’t exist, and horrible looping pattern can be seen. I don’t know what to do to fix it. The proper fix for it is to run:
[code]sudo service FOGMulticastManager stop
sudo service FOGImageReplicator stop
sudo service FOGScheduler stop
sudo service FOGMulticastManager start
sudo service FOGImageReplicator start
sudo service FOGScheduler start[/code]

gwhitfield

Thanks, Tom, I hadn’t come across the MySQL thing but I’ll make sure we’re doing the start-stop thing right from here on out.

Mentaloid

[quote=“Tom Elliott, post: 34771, member: 7271”]There’s a problem, similar to the tftpd-hpa issue, with mysql. I believe this is what you’re encountering.

It’s trying to start mysql, tftpd-hpa, and the FOG Services before the network device is available to run. What ends up happening is the processes (FOGMulticastManager,FOGImageReplicator,FOGScheduler) are running, but can’t communicate. So a restart doesn’t work because the restart simply kills and starts a new process. The previous process can’t stop so it just starts another process using the original process generated as if it were a child process. Because that doesn’t exist, and horrible looping pattern can be seen. I don’t know what to do to fix it. The proper fix for it is to run:
[code]sudo service FOGMulticastManager stop
sudo service FOGMulticastManager stop
sudo service FOGScheduler stop
sudo service FOGMulticastManager start
sudo service FOGImageReplicator start
sudo service FOGScheduler start[/code][/quote]

So, after taking a 5 minute look at this, there’s a quick fix, albeit an semi-incorrect fix.
Things to note here:
A) I use Debian.
B) The fog Ubuntu/Debian installer uses sys-rc-conf to register the initscripts which on a Debian system, while it works, is wrong.

The solution I came up with to make this work in a hurry:
A) Edit /etc/init.d/FOG* and change line #4 from [code]# Required-Start: $local_fs $remote_fs $network $syslog $network $inetd[/code] to [code]# Required-Start: $local_fs $remote_fs $network $syslog $network $inetd $all[/code]. Adding the $all keyword here is frowned upon, however I’m looking for a quick fix, not a correct fix… Possibly I will revert this at some point after changing the services themselves to start gracefully and wait for a network to be available, rather than bombing out. (If I do this, I’ll submit a patch)

B) update-rc.d (or insserv) is the correct way to register init scripts for Debian. I ran
[code]
insserv -d /etc/init.d/FOGMulticastManager
insserv -d /etc/init.d/FOGScheduler
insserv -d /etc/init.d/FOGImageReplicator
[/code] to reset the registration of the initscripts to the correct dependencies/runlevels as requested in the LSB headers of the initscripts.

After a reboot, all services were running, and I could get multicast running correctly.

Mentaloid

So I’ve been playing around more with this.
I’m using a fresh install of FOG 1.2.0 on Debian 7.6

First off, the init scripts don’t seem to be added to the startup correctly.
Fixed this via
[code]
insserv -d /etc/init.d/FOGMulticastManager
insserv -d /etc/init.d/FOGScheduler
insserv -d /etc/init.d/FOGImageReplicator
[/code]

Second, going by the old idea that the network wasn’t ready yet, I wrote a routine to wait for the network interface
/var/www/fog/commons/nettest.php
[code]
<?php
function clear_screen($outputdevice) { $GLOBALS[‘FOGCore’]->out(chr(27).“[2J”.chr(27).“[;H”,$outputdevice); }
function wait_interface_ready($interface,$outputdevice) {
while (true) {
$retarr = array();
exec(‘netstat -inN’,$retarr);
array_shift($retarr);array_shift($retarr);
foreach($retarr as $line) {
$t = substr($line,0,strpos($line,’ '));
if ($t === $interface) {
$GLOBALS[‘FOGCore’]->out(“Interface now ready…”,$outputdevice);
break 2;
}
}
$GLOBALS[‘FOGCore’]->out(“Interface not ready, waiting…”,$outputdevice);
sleep(10);
}
}
?>
[/code]
Second I added this to /opt/fog/service/FOG*/FOG* similar to below (with appropriate differences for the different files)
[code]
<?php
@error_reporting(0);
require_once( dirname(realpath(FILE)) . “/…/etc/config.php” );
require_once( WEBROOT . “/commons/base.inc.php” );

new code here

require_once( WEBROOT . “/commons/nettest.php” );
clear_screen(MULTICASTDEVICEOUTPUT);
$FOGCore->out($FOGCore->getBanner(), MULTICASTDEVICEOUTPUT);
wait_interface_ready($FOGCore->getSetting(‘FOG_UDPCAST_INTERFACE’),MULTICASTDEVICEOUTPUT);

    $MM = new MulticastManager();
    if( ! file_exists( UDPSENDERPATH ) )
    {
            $MM->outall(sprintf(" * Unable to locate udp-sender!."));
            exit;
    }
    $MM->serviceStart();
    $MM->serviceRun();
    $MM->outall(sprintf(" * Service has ended."));

?>
[/code]
At this point I tested with the network interfaces being up/down when service is started, and it tests ok… the various /opt/fog/service/FOG*/FOG* scripts will wait until a network device is available before continuing.
Now I rebooted, expecting the issue to be solved… but nope. While more robust, it’s still failing to start with a boot.
After fooling around awhile, I figured out that it’s because the FOG* scripts are being called before mysql is started in the boot sequence. These scripts fail hard without a SQL connection. I’m not sure how to fix that, but I’m not sure that it’s a needed fix at this point.
With the above routine still in place, I also modified the LSB headers in /etc/init.d/FOG*, removing the default run levels so the system can maintain them properly on it’s own, and adding mysql to the required start line (Yes, the lack of $ is important, as it’s a service provided by another init script, not a system service)
[code]

BEGIN INIT INFO

Provides: FOGMulticastManager

Required-Start: $local_fs $remote_fs $network $syslog $network $inetd mysql

Required-Stop: $local_fs $remote_fs $network $syslog $network $inetd

Default-Start:

Default-Stop:

X-Interactive: true

Short-Description: Start/Stop FOGMulticastManager

Long-Description: Created by Chuck Syperski

Used to stop and start the FOGMulticastManager Service.

FOGMulticastManager is used to destribute images through

Multicast. Useful to image large amounts of systems simultaneously.

It serves this ability only if it’s the master node.

END INIT INFO

[/code]
Ran insserv again to update the bootscripts…
[code]
insserv -d /etc/init.d/FOGMulticastManager
insserv -d /etc/init.d/FOGScheduler
insserv -d /etc/init.d/FOGImageReplicator
[/code]

And now after a reboot, all the services are starting correctly.

Tom Elliott

[quote=“Mentaloid, post: 35670, member: 4362”]So I’ve been playing around more with this.
I’m using a fresh install of FOG 1.2.0 on Debian 7.6

First off, the init scripts don’t seem to be added to the startup correctly.
Fixed this via
[code]
insserv -d /etc/init.d/FOGMulticastManager
insserv -d /etc/init.d/FOGScheduler
insserv -d /etc/init.d/FOGImageReplicator
[/code]

Second, going by the old idea that the network wasn’t ready yet, I wrote a routine to wait for the network interface
/var/www/fog/commons/nettest.php
[code]
<?php
function clear_screen($outputdevice) { $GLOBALS[‘FOGCore’]->out(chr(27).“[2J”.chr(27).“[;H”,$outputdevice); }
function wait_interface_ready($interface,$outputdevice) {
while (true) {
$retarr = array();
exec(‘netstat -inN’,$retarr);
array_shift($retarr);array_shift($retarr);
foreach($retarr as $line) {
$t = substr($line,0,strpos($line,’ '));
if ($t === $interface) {
$GLOBALS[‘FOGCore’]->out(“Interface now ready…”,$outputdevice);
break 2;
}
}
$GLOBALS[‘FOGCore’]->out(“Interface not ready, waiting…”,$outputdevice);
sleep(10);
}
}
?>
[/code]
Second I added this to /opt/fog/service/FOG*/FOG* similar to below (with appropriate differences for the different files)
[code]
<?php
@error_reporting(0);
require_once( dirname(realpath(FILE)) . “/…/etc/config.php” );
require_once( WEBROOT . “/commons/base.inc.php” );

new code here

require_once( WEBROOT . “/commons/nettest.php” );
clear_screen(MULTICASTDEVICEOUTPUT);
$FOGCore->out($FOGCore->getBanner(), MULTICASTDEVICEOUTPUT);
wait_interface_ready($FOGCore->getSetting(‘FOG_UDPCAST_INTERFACE’),MULTICASTDEVICEOUTPUT);

    $MM = new MulticastManager();
    if( ! file_exists( UDPSENDERPATH ) )
    {
            $MM->outall(sprintf(" * Unable to locate udp-sender!."));
            exit;
    }
    $MM->serviceStart();
    $MM->serviceRun();
    $MM->outall(sprintf(" * Service has ended."));

?>
[/code]
At this point I tested with the network interfaces being up/down when service is started, and it tests ok… the various /opt/fog/service/FOG*/FOG* scripts will wait until a network device is available before continuing.
Now I rebooted, expecting the issue to be solved… but nope. While more robust, it’s still failing to start with a boot.
After fooling around awhile, I figured out that it’s because the FOG* scripts are being called before mysql is started in the boot sequence. These scripts fail hard without a SQL connection. I’m not sure how to fix that, but I’m not sure that it’s a needed fix at this point.
With the above routine still in place, I also modified the LSB headers in /etc/init.d/FOG*, removing the default run levels so the system can maintain them properly on it’s own, and adding mysql to the required start line (Yes, the lack of $ is important, as it’s a service provided by another init script, not a system service)
[code]

BEGIN INIT INFO

Provides: FOGMulticastManager

Required-Start: $local_fs $remote_fs $network $syslog $network $inetd mysql

Required-Stop: $local_fs $remote_fs $network $syslog $network $inetd

Default-Start:

Default-Stop:

X-Interactive: true

Short-Description: Start/Stop FOGMulticastManager

Long-Description: Created by Chuck Syperski

Used to stop and start the FOGMulticastManager Service.

FOGMulticastManager is used to destribute images through

Multicast. Useful to image large amounts of systems simultaneously.

It serves this ability only if it’s the master node.

END INIT INFO

[/code]
Ran insserv again to update the bootscripts…
[code]
insserv -d /etc/init.d/FOGMulticastManager
insserv -d /etc/init.d/FOGScheduler
insserv -d /etc/init.d/FOGImageReplicator
[/code]

And now after a reboot, all the services are starting correctly.[/quote]

Mentaloid,

I’ve added much of the scripts you did (less the nettest.php as I added it directly into FOGCore class.) Also all the edits. Hopefully this will help us all out. Thank you for taking the time to test/troubleshoot and post your findings.

Mentaloid

[quote=“Tom Elliott, post: 35689, member: 7271”]Mentaloid,

I’ve added much of the scripts you did (less the nettest.php as I added it directly into FOGCore class.) Also all the edits. Hopefully this will help us all out. Thank you for taking the time to test/troubleshoot and post your findings.[/quote]

No worries at at all Tom, the least I can do to help the project!
I had figured you would add it directly to the FOGCore class

I tested this against Ubuntu as well. Unfortunately, Ubuntu/upstart ignores LSB headers in the init scripts (bah… Ubuntu re-invents the wheel, and cripples it in the process again). Another solution will have to be found for this mysql issue for more compatibility.
2 options exist…
A) write a routine that checks for mysql to be alive, similar to wait_interface_ready.
B) Create an upstart style initscript, and install that instead of insserv/lsb initscript when installing on upstart OS’s

I like option A better, as it wouldn’t require separating the installer into ubuntu/upstart and debian/inserv again (yuck). I’ll look into how/where you implemented wait_interface_ready in the FOGCore class, and see if I can create a similar routine for mysql this weekend (or soon thereafter). The problem I can see with this is that FOGCore itself is what bombs without mysql. I’ll toy and see whats possible.

I also noticed a device output mistake in /var/www/fog/lib/fog/MulticastManager.class.php on line 227. This should be MULTICASTDEVICEOUTPUT. In SVN, it is using REPLICATORDEVICEOUTPUT.

Tom Elliott

Thanks for reporting and I’ll fix it right now.

Mentaloid

Toying around complete… Conclusions:
A) once base.inc.php is included, the full FOGCore/FOGBase is initialized. There doesn’t look to be any way I can see with the current structure to have the classes wait for the database when used as a daemon. Once called, an exception and exit is desired for normal circumstances (called via the CGI/Apache SAPI) if the database isn’t available. If we create an exemption for daemons (CLI SAPI) during DB class construction, then we are stuck with having to determine which service is currently executing and output to the correct output device the daemon for messages and waiting. I don’t see this as desirable operation inside the base DB classes as this isn’t the job of the DB class, or for that matter FOGCore/Base.

B) have a class specific to daemons that handles wait_interface_ready and wait_db_ready routines. Feasibly, the only use for these routines would be for the daemons anyways.

I’ve gone ahead and have done option B.

Attached are the patched /opt/fog/service/FOG*/FOG* files, as well as a Daemon.class.php (/var/www/fog/lib/fog/Daemon.class.php).
$Daemon->wait_db_ready() and $Daemon->wait_interface_ready() are both implemented. wait_interface_ready() has some fixes to make it a little more compatible with servers that are using Network-Manager/Network-Manager-Gnome.

Tested against Debian 7.6 and Ubuntu 14.04, with and without NM purged.

[url=“/_imported_xf_attachments/1/1313_FOGServices.zip?:”]FOGServices.zip[/url]

Fog 1.1.0 multicast sits at "Starting to restore image (-) to device (/dev/sda1)

… from CLI I tried to restart the FOGMulticastManager and got this:

… so thinking “AHA!! The service wasn’t even running” I tried several times to start the service and got this:

… after thinking on it a bit I try to stop and start as individual steps:

… Oooooooh, that looks good so I try another restart to see what happens:

new code here

BEGIN INIT INFO

Provides: FOGMulticastManager

Required-Start: $local_fs $remote_fs $network $syslog $network $inetd mysql

Required-Stop: $local_fs $remote_fs $network $syslog $network $inetd

Default-Start:

Default-Stop:

X-Interactive: true

Short-Description: Start/Stop FOGMulticastManager

Long-Description: Created by Chuck Syperski

Used to stop and start the FOGMulticastManager Service.

FOGMulticastManager is used to destribute images through

Multicast. Useful to image large amounts of systems simultaneously.

It serves this ability only if it’s the master node.

END INIT INFO

new code here

BEGIN INIT INFO

Provides: FOGMulticastManager

Required-Start: $local_fs $remote_fs $network $syslog $network $inetd mysql

Required-Stop: $local_fs $remote_fs $network $syslog $network $inetd

Default-Start:

Default-Stop:

X-Interactive: true

Short-Description: Start/Stop FOGMulticastManager

Long-Description: Created by Chuck Syperski

Used to stop and start the FOGMulticastManager Service.

FOGMulticastManager is used to destribute images through

Multicast. Useful to image large amounts of systems simultaneously.

It serves this ability only if it’s the master node.

END INIT INFO

273

11.6k

17.1k

154.6k