Multicast does not work anymore

Tywyn

Hi there,

trying to deploy images to 2 machines via multicast, but it does not start. It keeps stopping at the Partclone-page.

Already cleared the DB with

TRUNCATE TABLE multicastSessions; 
TRUNCATE TABLE multicastSessionsAssoc; 
DELETE FROM tasks WHERE taskTypeId=8;

but this did not help.

/opt/fog/log/multicast.log says This is not the master node, but I have only one node installed.

Any ideas?

Tywyn

So, now I did some tests. With all changes I everytime restarted the FOGMulticastManager.

ens33 remained the same, having ip 10.203.8.31

Changed enp3s1 to 10.203.80.4 -> Multicast workes
Deactived enp3s1 -> Multicast workes
Changed enp3s1 to 192.168.100.1 (same config, which did not work before) -> Multicast workes
Changed enp3s1 to 10.203.80.4 -> Multicast workes

So, in all 4 cases Multicast works now. Very strange.
I would consider the case closed for me, but it is anyway strange.

Maybe someone could try to have same scenario as I had to maybe reproduce the thing. In brief:

Version 1.4.4
Setup 2 NIC with IP-addresses A and B
define IP Address A for fog-server -> multicast worked
Change fog-server address to B -> multicast stopped working
Chance IP A into IP C -> multicast works again.

Thank you for you help, gentlemen!

Sebastian Roth

@Tywyn Restared the FOGMulticast service or the whole server yet?

Tywyn

@Sebastian-Roth First tried to restart the FOGMulticastManager (then the logfile said, that it is already running?) and then I restartet the whole server a couple of times.

Sebastian Roth

@Tywyn Has this ever worked for you on this node? Just wondering what happened?!

Have you ever changed the servers IP at some point in time? Maybe this hasn’t been reflected in all the places needed?

Other than that you might want to take pictures of your Storage setup in the web UI and post here so we can have a look of there is something.

Tywyn

@Sebastian-Roth Good morning. Yes, this has worked and yes, there was a change in the IP-address quite a while ago. And it worked also after the change in the ip. I am not aware that I changed anything inbetween.

Sebastian Roth

@Tywyn So I suspect the IP we see in the picture is correct!? As well is the interface? To check on the FOG server run sudo ip a s dev ens33

Settings look ok from my point of view. Could you post more of what you have in the multicast.log. Maybe this will shed a light on what’s wrong.

By the way, which version of FOG do you use? Looks like 1.4.4, right?

Tywyn

@Sebastian-Roth : Hiya,

ens33 is correct and the ip 10.203.8.31 is also correct.
Running version 1.4.4, yes.

2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:25:11:5e:6a:86 brd ff:ff:ff:ff:ff:ff
    inet 10.203.8.31/21 brd 10.203.15.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fd00::4a79:ee8:ead6:2c9c/64 scope global noprefixroute dynamic 
       valid_lft 7110sec preferred_lft 3510sec
    inet6 fe80::78d3:810d:4c36:e801/64 scope link 
       valid_lft forever preferred_lft forever

mutlicast.log (Guess here is the error. Have had 192.168.100.1 before as IP):

== Released under GPL Version 3 ==
==================================


[11-29-17 2:00:45 pm] Interface Ready with IP Address: 127.0.0.1
[11-29-17 2:00:45 pm] Interface Ready with IP Address: 127.0.1.1
[11-29-17 2:00:45 pm] Interface Ready with IP Address: 192.168.100.1
[11-29-17 2:00:45 pm]  * Starting MulticastManager Service
[11-29-17 2:00:45 pm]  * Checking for new items every 10 seconds
[11-29-17 2:00:45 pm]  * Starting service loop
[11-29-17 2:00:45 pm]  | This is not the master node
[11-29-17 2:00:55 pm]  | This is not the master node
[11-29-17 2:01:05 pm]  | This is not the master node
[11-29-17 2:01:15 pm]  | This is not the master node
[11-29-17 2:01:25 pm]  | This is not the master node
[11-29-17 2:01:35 pm]  | This is not the master node

Where could be this 192.168.100.1 still configured?
Thanx for your help

PS: Will be out of office till next monday. So do not wonder, why I do not reply

Sebastian Roth

@Tywyn FOG actually enumerates the IP addresses. So please run /sbin/ip -4 addr | awk -F'[ /]+' '/global/ {print $3}' and /sbin/ifconfig -a | awk -F'[ /:]+' '/(cast)/ {print $4}' and post output here.

Maybe the network interface with the old IP is still up and running. Though I still wonder why FOG wouldn’t find the ens33 one.

Wayne Workman

@tywyn troubleshooting dictates we remove fog from the picture entirely and just test multicast on your network to determine if that works or not. Follow these instructions to do that https://wiki.fogproject.org/wik:i/index.php?title=Troubleshoot_Downloading_-_Multicast#Testing_Multicast
Ask us for help if you need it.

Tywyn

@wayne-workman

Will run through this troubleshooting page. Have to install a second linux first

Tywyn

Ok, setup a second linux machine and did the tests, that are described here:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_Downloading_-_Multicast

This test worked (I received on the second machine the fogsettings-file).

Still the normal deploy via multicast stopps at the clonepart-screen.

Tywyn

@sebastian-roth said in Multicast does not work anymore:

/sbin/ifconfig -a | awk -F’[ /:]+’ ‘/(cast)/ {print $4}’

# /sbin/ip -4 addr | awk -F'[ /]+' '/global/ {print $3}'
10.203.8.31
192.168.100.1

Yes, the old interface is still running. From the beginning we have had both ip-addresses. I just switched fog from one interface to the other one.

The other command you postet just output twice “netmask”. So here is the full ifconfig.

# /sbin/ifconfig -a 
enp3s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.1  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::a61b:9cdd:69ce:1d4b  prefixlen 64  scopeid 0x20<link>
        ether 98:de:d0:1b:bd:d5  txqueuelen 1000  (Ethernet)
        RX packets 83  bytes 7640 (7.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 180  bytes 17066 (16.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.203.8.31  netmask 255.255.248.0  broadcast 10.203.15.255
        inet6 fd00::4a79:ee8:ead6:2c9c  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::78d3:810d:4c36:e801  prefixlen 64  scopeid 0x20<link>
        ether 00:25:11:5e:6a:86  txqueuelen 1000  (Ethernet)
        RX packets 57921  bytes 7461829 (7.1 MiB)
        RX errors 0  dropped 13  overruns 0  frame 0
        TX packets 6536  bytes 1220003 (1.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Lokale Schleife)
        RX packets 1156  bytes 162929 (159.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1156  bytes 162929 (159.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Sebastian Roth

@Tywyn What kind of Linux OS do you run? Seems like the ifconfig output is a different format to what we are used to.

Tywyn

@Sebastian-Roth
It is a debian 9.

Sebastian Roth

@tywyn Alright, so it seems like the ifconfig output format changed. But FOG uses the /sbin/ip -4 ... command first and so it shouldn’t be an issue.

I have looked through the code thoroughly but still can’t think of why we only see 192.168.100.1 in the multicast log. We should see the other one as well! Possibly I’ve just looked it over and need to get a rest. Maybe someone else can see what I am missing here.

Wayne Workman

@sebastian-roth We need to clearly define the problem. As I see it, there’s two local interfaces on this FOG Server.

enp3s1 @ 192.168.100.1
ens33 @ 10.203.8.31

And we are seeing multicast try to use enp3s1 instead of the one we see defined on the storage node in the picture ens33.

I remember @Tom-Elliott doing work on this, something about auto-selecting the interface if I remember right. But really, the interface that Multicast should use ought to be the defined interface in storage node settings.

@Tywyn you could try to turn the interface off and see if multicast works then? Try this command: ip link set enp3s1 down You can later turn it back on with ip link set enp3s1 up also a reboot would bring it back up too.

Tywyn

So, now I did some tests. With all changes I everytime restarted the FOGMulticastManager.

ens33 remained the same, having ip 10.203.8.31

Changed enp3s1 to 10.203.80.4 -> Multicast workes
Deactived enp3s1 -> Multicast workes
Changed enp3s1 to 192.168.100.1 (same config, which did not work before) -> Multicast workes
Changed enp3s1 to 10.203.80.4 -> Multicast workes

So, in all 4 cases Multicast works now. Very strange.
I would consider the case closed for me, but it is anyway strange.

Maybe someone could try to have same scenario as I had to maybe reproduce the thing. In brief:

Version 1.4.4
Setup 2 NIC with IP-addresses A and B
define IP Address A for fog-server -> multicast worked
Change fog-server address to B -> multicast stopped working
Chance IP A into IP C -> multicast works again.

Thank you for you help, gentlemen!

Sebastian Roth

@Tywyn Great to hear you could make it work again. I have no idea what was wrong before. I have looked through the code many times and don’t know what could have gone wrong. Marking this solved now. Let us know if you see this again.

Multicast does not work anymore

243

11.7k

17.2k

154.7k