Multicasting Issues

kenneth.sisco · Jan 9, 2016, 6:59 PM

I’m having a lot of trouble getting multicast working in my environment. I work for a University where our networks are all segmented into multiple VLANs, and I don’t have the ability to put FOG on the same VLAN as the clients - we have about 10 computer labs and our FOG server is a VMWare virtual machine in our data center. Each computer lab is on its own VLAN, and the data center is as well.

Our network team has a lot of controls on the network that I cannot see, but they assigned me a multicast range of 239.132.20.0 /24. Since this doesn’t match the default that FOG uses, I upgraded to the Trunk version of FOG to allow me to customize that. The network team setup a test multicast node within that multicast range and ran several small tests from their utilities across all of my VLANs and have had no hiccups at all, leading them to believe that the problem lies on the FOG server somewhere.

Here’s where I’m at on the server:
OS: Ubuntu Server 14.04.3 LTS
FOG Trunk revision: 4623

On the FOG server, I have a small 8GB image for testing. Under FOG Configuration -> FOG Settings -> Multicast Settings, I have configured the following:

FOG_UDPCAST_INTERFACE: eth0
FOG_UDPCAST_STARTINGPORT: 63072
FOG_MULTICAST_MAX_SESSIONS: 64
FOG_UDPCAST_MAXWAIT	: 10
FOG_MULTICAST_ADDRESS: 239.132.20.10
FOG_MULTICAST_PORT_OVERRIDE: 0
FOG_MULTICAST_DUPLEX: FULL_DUPLEX

I have registered two test hosts, and am attempting to get one working before trying two. I’ve double-checked under Active Tasks and Active Multicast tasks to ensure nothing is currently running. I’ve rebooted the server and restarted FOGMulticastManager, FOGImageReplicator, and FOGScheduler services (in case they didn’t start properly).

When trying to multicast to a single client, I get inconsistent results. Most of the time, the client gets stuck at “Starting to restore image ( - ) to device (/dev/sda1).” Other times, it’ll get past that point and show a status bar, but the bar will be stuck at 0% with a transfer rate of 0.00bytes/min. The latter of which is incredibly difficult for me to replicate, I’ve found. Most of the time, I’m stuck at the former.

Here’s what the multicast.log looks like:

[01-09-16 6:31:41 pm] Interface Ready with IP Address: 10.133.132.127

[01-09-16 6:31:41 pm]  * Starting MulticastManager Service

[01-09-16 6:31:41 pm]  * Checking for new items every 10 seconds

[01-09-16 6:31:41 pm]  * Starting service loop

[01-09-16 6:31:41 pm]  * No tasks found!
[01-09-16 6:31:51 pm]  * No tasks found!

[01-09-16 6:32:01 pm]  * No tasks found!

[01-09-16 6:32:11 pm]  | Sleeping for 10 seconds to ensure tasks are properly submitted

[01-09-16 6:32:21 pm]  | 0 tasks to be cleaned
[01-09-16 6:32:21 pm]  | 1 task found

[01-09-16 6:32:21 pm]  | Task (11) Multi-Cast Task is new!

[01-09-16 6:32:21 pm]  | Task (11) Multi-Cast Task has been cleaned.

[01-09-16 6:32:21 pm]  | Task (11) /images/Test162016 image file found.

[01-09-16 6:32:21 pm]  | Task (11) 1 client(s) found.
Udp-sender 20120424

[01-09-16 6:32:21 pm]  | Task (11) Multi-Cast Task sending on base port: 64512
UDP sender for (stdin) at 10.133.132.127 on eth0 
Broadcasting control to 224.0.0.1
[01-09-16 6:32:21 pm]  | CMD: cat /images/Test162016/d1p1.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 1 --max-wait 600 --mcast-data-address 239.132.20.10 --portbase 64512 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/Test162016/d1p2.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 1 --max-wait 600 --mcast-data-address 239.132.20.10 --portbase 64512 --full-duplex --ttl 32 --nokbd --nopointopoint;
[01-09-16 6:32:21 pm]  | Task (11) Multi-Cast Task has started.

[01-09-16 6:32:31 pm]  | 0 tasks to be cleaned

[01-09-16 6:32:31 pm]  | 1 task found

[01-09-16 6:32:31 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:32:41 pm]  | 0 tasks to be cleaned

[01-09-16 6:32:41 pm]  | 1 task found

[01-09-16 6:32:41 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:32:51 pm]  | 0 tasks to be cleaned

[01-09-16 6:32:51 pm]  | 1 task found

[01-09-16 6:32:51 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:33:01 pm]  | 0 tasks to be cleaned
[01-09-16 6:33:01 pm]  | 1 task found

[01-09-16 6:33:01 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:33:11 pm]  | 0 tasks to be cleaned

[01-09-16 6:33:11 pm]  | 1 task found

[01-09-16 6:33:11 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:33:21 pm]  | 0 tasks to be cleaned

[01-09-16 6:33:21 pm]  | 1 task found

[01-09-16 6:33:21 pm]  | Task (11) Multi-Cast Task is already running PID 3793
New connection from 10.20.129.47  (#0) 00000009
Starting transfer: 00000009
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:33:31 pm]  | 0 tasks to be cleaned

[01-09-16 6:33:31 pm]  | 1 task found

[01-09-16 6:33:31 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:33:41 pm]  | 0 tasks to be cleaned

[01-09-16 6:33:41 pm]  | 1 task found

[01-09-16 6:33:41 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:33:51 pm]  | 0 tasks to be cleaned

[01-09-16 6:33:51 pm]  | 1 task found

[01-09-16 6:33:51 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:34:02 pm]  | 0 tasks to be cleaned

[01-09-16 6:34:02 pm]  | 1 task found

[01-09-16 6:34:02 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:34:12 pm]  | 0 tasks to be cleaned
[01-09-16 6:34:12 pm]  | 1 task found

[01-09-16 6:34:12 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:34:22 pm]  | 0 tasks to be cleaned

[01-09-16 6:34:22 pm]  | 1 task found

[01-09-16 6:34:22 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:34:32 pm]  | 0 tasks to be cleaned

[01-09-16 6:34:32 pm]  | 1 task found

[01-09-16 6:34:32 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:34:42 pm]  | 0 tasks to be cleaned

[01-09-16 6:34:42 pm]  | 1 task found

[01-09-16 6:34:42 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:34:52 pm]  | 0 tasks to be cleaned

[01-09-16 6:34:52 pm]  | 1 task found

[01-09-16 6:34:52 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:35:02 pm]  | 0 tasks to be cleaned

[01-09-16 6:35:02 pm]  | 1 task found

[01-09-16 6:35:02 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:35:12 pm]  | 0 tasks to be cleaned

[01-09-16 6:35:12 pm]  | 1 task found

[01-09-16 6:35:12 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:35:22 pm]  | 0 tasks to be cleaned

[01-09-16 6:35:22 pm]  | 1 task found

[01-09-16 6:35:22 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:35:32 pm]  | 0 tasks to be cleaned

[01-09-16 6:35:32 pm]  | 1 task found

[01-09-16 6:35:32 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:35:42 pm]  | 0 tasks to be cleaned

[01-09-16 6:35:42 pm]  | 1 task found

[01-09-16 6:35:42 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:35:52 pm]  | 0 tasks to be cleaned

[01-09-16 6:35:52 pm]  | 1 task found

[01-09-16 6:35:52 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:36:02 pm]  | 0 tasks to be cleaned

[01-09-16 6:36:02 pm]  | 1 task found

[01-09-16 6:36:02 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:36:12 pm]  | 0 tasks to be cleaned

[01-09-16 6:36:12 pm]  | 1 task found

[01-09-16 6:36:12 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:36:22 pm]  | 0 tasks to be cleaned

[01-09-16 6:36:22 pm]  | 1 task found

[01-09-16 6:36:22 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:36:32 pm]  | 0 tasks to be cleaned

[01-09-16 6:36:32 pm]  | 1 task found

[01-09-16 6:36:32 pm]  | Task (11) Multi-Cast Task is already running PID 3793

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000

[01-09-16 6:36:42 pm]  | 0 tasks to be cleaned

[01-09-16 6:36:42 pm]  | 1 task found

[01-09-16 6:36:42 pm]  | Task (11) Multi-Cast Task is already running PID 3793
Dropping client #0 because of timeout
Disconnecting #0 (10.20.129.47)
bytes=                re-xmits=0000000 (  0.0%) slice=0112 -   0
Transfer complete.

cat: write error: Broken pipe
Udp-sender 20120424
UDP sender for (stdin) at 10.133.132.127 on eth0 
Broadcasting control to 224.0.0.1

[01-09-16 6:36:52 pm]  | 0 tasks to be cleaned

[01-09-16 6:36:52 pm]  | 1 task found

[01-09-16 6:36:52 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:37:02 pm]  | 0 tasks to be cleaned

[01-09-16 6:37:02 pm]  | 1 task found

[01-09-16 6:37:02 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:37:12 pm]  | 0 tasks to be cleaned

[01-09-16 6:37:12 pm]  | 1 task found

[01-09-16 6:37:12 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:37:22 pm]  | 0 tasks to be cleaned

[01-09-16 6:37:22 pm]  | 1 task found

[01-09-16 6:37:22 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:37:32 pm]  | 0 tasks to be cleaned

[01-09-16 6:37:32 pm]  | 1 task found

[01-09-16 6:37:32 pm]  | Task (11) Multi-Cast Task is already running PID 3793

[01-09-16 6:37:42 pm]  | 0 tasks to be cleaned

[01-09-16 6:37:42 pm]  | 1 task found

[01-09-16 6:37:42 pm]  | Task (11) Multi-Cast Task is already running PID 3793
[01-09-16 6:37:52 pm]  | 0 tasks to be cleaned

[01-09-16 6:37:52 pm]  | 1 task found

[01-09-16 6:37:52 pm]  | Task (11) Multi-Cast Task is already running PID 3793

I’m not sure where to go next in my troubleshooting. Any help?

Wayne Workman · Jan 9, 2016, 7:40 PM

@kenneth.sisco Can FOG Multicast on the VLAN that it’s actually within ? We should focus on that first.

Tom Elliott · Jan 9, 2016, 8:59 PM

Most times when I see the “This task has been cleaned” directly after it starting, I find it is because of the interface of the network not being set properly for the interface on the system trying to run the multicast job. I will test just to see if things are still good, but I’m nearly certain this is the case.

You could try running the command you see in the log by itself to see if it spits out any other pertinent information as well.

kenneth.sisco · Jan 9, 2016, 11:32 PM

@Tom-Elliott said:

Most times when I see the “This task has been cleaned” directly after it starting, I find it is because of the interface of the network not being set properly for the interface on the system trying to run the multicast job. I will test just to see if things are still good, but I’m nearly certain this is the case.

You could try running the command you see in the log by itself to see if it spits out any other pertinent information as well.

I ran the command by hand and didn’t see anything happen. Booting the client to test the multicast didn’t change this. Logs looked exactly the same as what I posted above. Here’s a screenshot of what I saw when I type the command by hand:

@Wayne-Workman said:

@kenneth.sisco Can FOG Multicast on the VLAN that it’s actually within ? We should focus on that first.

I wish I could test that, but the network team doesn’t allow DHCP to be enabled on server VLANs (or within the data center for that matter). If you know of a way I could do this without DHCP, I’d be happy to try.

Tom Elliott · Jan 10, 2016, 2:02 AM

@kenneth.sisco said:

sounds like, to me, the switch is not allowing udp broadcast forwarding. This is just a guess at this point.

I’ve tested Multicast, and all seems to be working fine.

I’ve also messed a lot, today, with the fog scripts. Hopefully things are better and I’ve been republishing them as I find things and fix them too.

To me, like Wayne suggested, you should start by imaging within the same switch first (not across vlan’s).

kenneth.sisco · Jan 10, 2016, 2:18 AM

I’ll reach out to our network team on Monday, then, and see if they’ll help me setup a test environment. Thanks for the help - I’ll report back soon!

george1421 · Jan 10, 2016, 2:24 PM

@kenneth.sisco Make sure that your network switches have igmp forwarding or snooping enabled. If you have vlans in your network the igmp settings are typically set on a per vlan basis. The igmp settings allows the switch ports to subscribe to the multicast channel. The multicast stream will only be sent to ports and switches that subscribe to the multicast request. This keeps multicast traffic off switches where there are no subscribers.

kenneth.sisco · Jan 11, 2016, 9:20 PM

So I worked with our network team and we were able to setup a small DHCP pool on the server VLAN temporarily for a test, and it worked fine. Once we tried again cross-VLAN, we had the same issues. I’ve copied the multicast.log section of the successful multicast below. The unsuccessful one is the same as my prior post in this thread. What I find interesting is that the multicast.log file shows that there is a new connection from the client cross-VLAN and even echoes its IP Address, but never transfers any data.

Successful multicast (same VLAN; shortened to allow for forum posting):

[01-11-16 4:54:24 pm]  * No tasks found!
[01-11-16 4:54:34 pm]  | Sleeping for 10 seconds to ensure tasks are properly submitted
[01-11-16 4:54:44 pm]  | 0 tasks to be cleaned
[01-11-16 4:54:44 pm]  | 1 task found
[01-11-16 4:54:44 pm]  | Task (14) Multi-Cast Task is new!
[01-11-16 4:54:44 pm]  | Task (14) Multi-Cast Task has been cleaned.
[01-11-16 4:54:44 pm]  | Task (14) /images/Test162016 image file found.
[01-11-16 4:54:44 pm]  | Task (14) 1 client(s) found.
[01-11-16 4:54:44 pm]  | Task (14) Multi-Cast Task sending on base port: 50702
Udp-sender 20120424
UDP sender for (stdin) at 10.133.132.127 on eth0 
Broadcasting control to 224.0.0.1
[01-11-16 4:54:44 pm]  | CMD: cat /images/Test162016/d1p1.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 1 --max-wait 600 --mcast-data-address 239.132.20.10 --portbase 50702 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/Test162016/d1p2.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 1 --max-wait 600 --mcast-data-address 239.132.20.10 --portbase 50702 --full-duplex --ttl 32 --nokbd --nopointopoint;
[01-11-16 4:54:44 pm]  | Task (14) Multi-Cast Task has started.
[01-11-16 4:54:54 pm]  | 0 tasks to be cleaned
[01-11-16 4:54:54 pm]  | 1 task found
[01-11-16 4:54:54 pm]  | Task (14) Multi-Cast Task is already running PID 27247
New connection from 10.133.132.242  (#0) 00000009
Starting transfer: 00000009
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000
[01-11-16 4:55:04 pm]  | 0 tasks to be cleaned
[01-11-16 4:55:04 pm]  | 1 task found
[01-11-16 4:55:04 pm]  | Task (14) Multi-Cast Task is already running PID 27247
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000
[01-11-16 4:55:14 pm]  | 0 tasks to be cleaned
[01-11-16 4:55:14 pm]  | 1 task found
[01-11-16 4:55:14 pm]  | Task (14) Multi-Cast Task is already running PID 27247
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000
[01-11-16 4:55:24 pm]  | 0 tasks to be cleaned
[01-11-16 4:55:24 pm]  | 1 task found
[01-11-16 4:55:24 pm]  | Task (14) Multi-Cast Task is already running PID 27247
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000
[01-11-16 4:55:34 pm]  | 0 tasks to be cleaned
[01-11-16 4:55:34 pm]  | 1 task found
[01-11-16 4:55:34 pm]  | Task (14) Multi-Cast Task is already running PID 27247
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000
[01-11-16 4:55:44 pm]  | 0 tasks to be cleaned
[01-11-16 4:55:44 pm]  | 1 task found
[01-11-16 4:55:44 pm]  | Task (14) Multi-Cast Task is already running PID 27247
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=10000
bytes=     131 040  re-xmits=0000090 (100.0%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=2143
bytes=  4 925 648  re-xmits=0000285 (  8.4%) slice=0112 -   0
bytes=  8 441 244  re-xmits=0000408 (  7.0%) slice=0112 -   0

Transfer complete.
Disconnecting #0 (10.133.132.242)

Udp-sender 20120424
UDP sender for (stdin) at 10.133.132.127 on eth0 
Broadcasting control to 224.0.0.1
[01-11-16 4:55:54 pm]  | 0 tasks to be cleaned
[01-11-16 4:55:54 pm]  | 1 task found
[01-11-16 4:55:54 pm]  | Task (14) Multi-Cast Task is already running PID 27247
New connection from 10.133.132.242  (#0) 00000009
[01-11-16 4:56:04 pm]  | 0 tasks to be cleaned
[01-11-16 4:56:04 pm]  | 1 task found
[01-11-16 4:56:04 pm]  | Task (14) Multi-Cast Task is already running PID 27247
Starting transfer: 00000009
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1606
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1554
bytes=  4 976 608  re-xmits=0000206 (  6.0%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=877
bytes= 33 904 416  re-xmits=0003153 ( 13.5%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=414
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1051
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1001
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=917
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=981
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1439
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1504
bytes= 51 679 264  re-xmits=0005546 ( 15.6%) slice=0112 -   0
bytes= 67 660 320  re-xmits=0009744 ( 20.9%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1059
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=732
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=553
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=576
bytes=102 883 872  re-xmits=0013016 ( 18.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=725
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=666
bytes=133 985 488  re-xmits=0015786 ( 17.1%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=837
bytes=154 532 560  re-xmits=0020475 ( 19.2%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=678
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=898
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=778
bytes=161 870 800  re-xmits=0021278 ( 19.1%) slice=0112 -   0
bytes=170 513 616  re-xmits=0021443 ( 18.3%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=546
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=577
bytes=181 951 952  re-xmits=0022699 ( 18.1%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=6126
bytes=203 314 384  re-xmits=0026621 ( 19.0%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=312
bytes=250 215 056  re-xmits=0027340 ( 15.9%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=415
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=492
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=597
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=543
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=619
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=717
bytes=279 522 880  re-xmits=0030935 ( 16.1%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=835
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=713
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=600
bytes=293 384 000  re-xmits=0034010 ( 16.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1189
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1052
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=943
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=949
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1505
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1134
[01-11-16 4:56:14 pm]  | 0 tasks to be cleaned
[01-11-16 4:56:14 pm]  | 1 task found
[01-11-16 4:56:14 pm]  | Task (14) Multi-Cast Task is already running PID 27247
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1066
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=959
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1214
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1660
bytes=302 842 176  re-xmits=0036928 ( 17.7%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1634
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1452
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1422
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1340
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1596
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1744
bytes=313 278 784  re-xmits=0039658 ( 18.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=569
bytes=333 988 928  re-xmits=0044769 ( 19.5%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=690
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=784
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=640
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=886
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=819
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=807
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=805
bytes=356 819 008  re-xmits=0049007 ( 19.9%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=816
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=709
bytes=378 670 656  re-xmits=0053067 ( 20.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=614
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=596
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=778
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=878
bytes=400 848 448  re-xmits=0057995 ( 21.0%) slice=0112 -   0
bytes=423 189 312  re-xmits=0062372 ( 21.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=696
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=924
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=644
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=652
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=695
bytes=444 225 600  re-xmits=0066687 ( 21.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=612
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=647
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=749
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=581
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=544
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=538
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=659
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=546
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=832
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=546
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=540
bytes=465 588 032  re-xmits=0071328 ( 22.3%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=821
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=810
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=857
bytes=484 341 312  re-xmits=0075419 ( 22.6%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1287
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1190
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1295
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1136
bytes=496 897 856  re-xmits=0078113 ( 22.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=945
bytes=518 097 216  re-xmits=0082459 ( 23.1%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=902
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1023
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1383
bytes=537 665 856  re-xmits=0086766 ( 23.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=718
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=752
bytes=561 637 440  re-xmits=0090529 ( 23.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=756
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=796
bytes=586 587 456  re-xmits=0094555 ( 23.4%) slice=0112 -   0
bytes=611 048 256  re-xmits=0098563 ( 23.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=912
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=668
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=596
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=610
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=716
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=540
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=540
bytes=636 324 416  re-xmits=0102313 ( 23.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=539
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=727
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=927
bytes=664 535 872  re-xmits=0105315 ( 23.0%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=746
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=499
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=751
bytes=688 670 528  re-xmits=0109292 ( 23.1%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=406
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=705
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=868
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=753
bytes=713 131 328  re-xmits=0113039 ( 23.0%) slice=0112 -   0
[01-11-16 4:56:24 pm]  | 0 tasks to be cleaned
[01-11-16 4:56:24 pm]  | 1 task found
[01-11-16 4:56:24 pm]  | Task (14) Multi-Cast Task is already running PID 27247

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=630
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=464
bytes=741 832 000  re-xmits=0115772 ( 22.7%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=724
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=721
bytes=770 206 528  re-xmits=0119096 ( 22.5%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=644
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=639
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1026
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=580
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=581
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=504
bytes=799 722 560  re-xmits=0121997 ( 22.2%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=623
bytes=826 140 224  re-xmits=0124891 ( 22.0%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=620
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=826
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=948
bytes=850 601 024  re-xmits=0129301 ( 22.1%) slice=0112 -   0
bytes=874 409 536  re-xmits=0134118 ( 22.3%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=661
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=560
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=638
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=548
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=667
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=549
bytes=900 174 912  re-xmits=0138525 ( 22.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=450
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=546
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=518
bytes=939 801 408  re-xmits=0140648 ( 21.7%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=924
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=755
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=767
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1026
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1114
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=723
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=712
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=652
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=595
bytes=954 640 960  re-xmits=0144344 ( 22.0%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1225
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=978
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1123
bytes=973 720 384  re-xmits=0147661 ( 22.0%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=523
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=599
bytes=     984 659K re-xmits=0150993 ( 21.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=674
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=531
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=593
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=638
bytes=  1 012 528K re-xmits=0153893 ( 21.6%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1122
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1069
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=996
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1102
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=742
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=646
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=713
bytes=  1 031 001K re-xmits=0156927 ( 21.6%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=968
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=951
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1503
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1364
bytes=  1 043 487K re-xmits=0159664 ( 21.7%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=449
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=886
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=703
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=684
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=660
bytes=  1 065 598K re-xmits=0162034 ( 21.6%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=567
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=543
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=594
bytes=  1 110 987K re-xmits=0163125 ( 20.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=311
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=357
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=809
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=764
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=759
bytes=  1 126 912K re-xmits=0164574 ( 20.7%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=472
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1227
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1171
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1080
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1242
bytes=  1 141 722K re-xmits=0167156 ( 20.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1113
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=856
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=718
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=469
bytes=  1 167 043K re-xmits=0171682 ( 20.9%) slice=0112 -   0
[01-11-16 4:56:34 pm]  | 0 tasks to be cleaned
[01-11-16 4:56:34 pm]  | 1 task found
[01-11-16 4:56:34 pm]  | Task (14) Multi-Cast Task is already running PID 27247
bytes=  1 201 919K re-xmits=0174903 ( 20.6%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=704
bytes=  1 225 966K re-xmits=0176769 ( 20.5%) slice=0112 -   0
bytes=  1 240 457K re-xmits=0178749 ( 20.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=866
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=736
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=878
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=935
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=892
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=779
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=748
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1026
bytes=  1 244 757K re-xmits=0179769 ( 20.5%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=516
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=736
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=913
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=829
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=873
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=939
bytes=  1 262 593K re-xmits=0183852 ( 20.7%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=651
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=708
bytes=  1 286 481K re-xmits=0188220 ( 20.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=553
bytes=  1 315 464K re-xmits=0192460 ( 20.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=836
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=765
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=524
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=530
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=537
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1177
bytes=  1 340 148K re-xmits=0196350 ( 20.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=666
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=747
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=774
bytes=  1 351 614K re-xmits=0198288 ( 20.8%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=448
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=851
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=881
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1157
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1428
bytes=  1 368 335K re-xmits=0201724 ( 20.9%) slice=0112 -   0
bytes=  1 392 063K re-xmits=0207147 ( 21.1%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=715
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=732
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=607
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=687
bytes=  1 414 677K re-xmits=0212032 ( 21.3%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1105
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1198
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=892
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=781
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=705
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=748
bytes=  1 432 513K re-xmits=0215533 ( 21.3%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=409
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=391
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=416
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=434
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=751
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=756
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=601
bytes=  1 462 770K re-xmits=0219103 ( 21.2%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=832
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=706
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=641
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=649
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=669
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=511
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=880
bytes=  1 485 702K re-xmits=0223774 ( 21.4%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=727
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=761
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=595
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=599
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=814
bytes=  1 507 997K re-xmits=0228263 ( 21.5%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=783
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=897
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=893
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=976
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=929
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=971
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=783
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1336
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=924
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=1032
[01-11-16 4:56:44 pm]  | 0 tasks to be cleaned
[01-11-16 4:56:44 pm]  | 1 task found
[01-11-16 4:56:44 pm]  | Task (14) Multi-Cast Task is already running PID 27247
bytes=  1 519 782K re-xmits=0231518 ( 21.6%) slice=0112 -   0

Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=636
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=596
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=633
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=636
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=607
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=640
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=638
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=690
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=822
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=516
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=601
Timeout notAnswered=[0] notReady=[0] nrAns=0 nrRead=0 nrPart=1 avg=998
bytes=  1 540 803K re-xmits=0236158 ( 21.7%) slice=0112 -   0


Transfer complete.
Disconnecting #0 (10.133.132.242)

[01-11-16 4:57:44 pm]  | 0 tasks to be cleaned
[01-11-16 4:57:44 pm]  | 1 task found
[01-11-16 4:57:44 pm]  | Task (14) Multi-Cast Task is no longer running.
[01-11-16 4:57:44 pm]  | Task (14) Multi-Cast Task has been cleaned as complete.
[01-11-16 4:57:54 pm]  * No tasks found!

Tom Elliott · Jan 11, 2016, 9:30 PM

@kenneth.sisco making a connection is simple but the udp stream is not being allowed to pass after the initial connection is made. The separated network method you used sounds to have a slight difference from the real network side. I don’t know what or where. This does not sound like something fog is failing to do though, like I said, I’d take a look if you can on whether broadcast forwarding is on.

Wayne Workman · Jan 11, 2016, 11:25 PM

@Tom-Elliott said:

I’d take a look if you can on whether broadcast forwarding is on.

I agree with Tom. Broadcast forwarding is exactly what you need. This happens on your router - as a routing device is the only way that data from one LAN (or V-LAN) can traverse to other LANs (or V-LANS).

Broadcast forwarding is called different things based on what vendor / piece of equipment you use. But it’s quite simply that which allows broadcast traffic to pass through a router. You’re network team don’t need to open everything up, they just need to allow passage from the server vlan to other vlans, and back. If your network team is really skilled, they can even limit it to a specific port on a specific switch, or a specific sending IP (the fog server’s IP).

kenneth.sisco · Jan 13, 2016, 4:03 AM

So I’m working with our network team to troubleshoot this, and we have “broadcast forwarding” working for the most part, but we’re going to keep digging. One thing we were concerned about was TTL on the client side. We can see the TTL from the server in the multicast.log, but we’re concerned about what the TTL is on the udp-receiver command. Is there a way to see that? Or a way to specify it? I was looking for the task files that 0.32 used to have stored in the tftpboot folder, but this version of Fog handles that differently, so I’m not sure where to find them.

Wayne Workman · Jan 13, 2016, 4:03 AM

@kenneth.sisco said:

Is there a way to see that? Or a way to specify it?

It’s set to 32.

The setting is part of the init, so to change it, you’ll need to edit it.

Here are instructions on editing the init:
https://wiki.fogproject.org/wiki/index.php?title=Modifying_the_Init_Image

Here’s the command I used to find the command and it’s output.

grep -rnw '/root/svn/trunk' -e "udp-receiver"

/root/svn/trunk/.svn/pristine/34/347286bf65c7493cf5cd96049b458c34ad245cff.svn-base:374:        udp-receiver --nokbd --portbase $port --ttl 32 --mcast-rdv-address $storageip 2>/dev/null > /tmp/pigz1 &
/root/svn/trunk/.svn/pristine/9f/9fbd1e42a40bc6fc567380e512a84c93d309d6b7.svn-base:442:            udp-receiver --nokbd --portbase $port --ttl 32 --mcast-rdv-address $storageip 2>/dev/null >/tmp/pigz1 &
/root/svn/trunk/src/buildroot/package/fog/scripts/usr/share/fog/lib/funcs.sh:442:            udp-receiver --nokbd --portbase $port --ttl 32 --mcast-rdv-address $storageip 2>/dev/null >/tmp/pigz1 &

For the record, I have went through 0.32’s code and I don’t see that ttl is specified at all in 0.32.
Here is the output of the above search command when I pointed it at the 0.32 directory I checked out via svn:

(the code lines are very wide, click and drag right to see it all)

/root/svn/0.32/trunk/.svn/pristine/0c/0ce1eec0cc580ce17d377e5ef7c3ca6b4d39abab.svn-base:271:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/.svn/pristine/0c/0ce1eec0cc580ce17d377e5ef7c3ca6b4d39abab.svn-base:273:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null |  partimage -f3 -b restore $win7rec stdin
/root/svn/0.32/trunk/.svn/pristine/0c/0ce1eec0cc580ce17d377e5ef7c3ca6b4d39abab.svn-base:274:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/.svn/pristine/0c/0ce1eec0cc580ce17d377e5ef7c3ca6b4d39abab.svn-base:278:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | partimage -f3 -b restore ${part} stdin
/root/svn/0.32/trunk/.svn/pristine/0c/0ce1eec0cc580ce17d377e5ef7c3ca6b4d39abab.svn-base:381:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | partimage -f3 -b restore $part stdin;
/root/svn/0.32/trunk/.svn/pristine/0c/0ce1eec0cc580ce17d377e5ef7c3ca6b4d39abab.svn-base:469:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null |  partimage -f3 -b restore $part stdin ;
/root/svn/0.32/trunk/.svn/pristine/0c/0ce1eec0cc580ce17d377e5ef7c3ca6b4d39abab.svn-base:519:                                                    udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | dd of=${hd}
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:340:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:342:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null |  partimage -f3 -b restore $win7rec stdin
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:343:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:346:                                                        #udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null |  partimage -f3 -b restore ${hd}1 stdin
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:347:                                                        #udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null |  partimage -f3 -b restore ${hd}2 stdin
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:348:                                                        #udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:352:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore ${part} stdin
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:467:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore $part stdin;
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:555:                                                        udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null |  partimage -f3 -b restore $part stdin ;
/root/svn/0.32/trunk/.svn/pristine/24/2461bc1e810a56b21208edfc4164f513270cdba2.svn-base:605:                                                    udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | dd of=${hd}
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:340:                                                                   udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:342:                                                                   udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null |  partimage -f3 -b restore $win7rec stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:343:                                                                   udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:346:                                                           #       udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null |  partimage -f3 -b restore ${hd}1 stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:347:                                                           #       udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null |  partimage -f3 -b restore ${hd}2 stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:348:                                                           #       udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:352:                                                           udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore ${part} stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:467:                                                                       udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | partimage -f3 -b restore $part stdin;
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:555:                                                                       udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null |  partimage -f3 -b restore $part stdin ;
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog:605:                                                   udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | dd of=${hd}
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog.bkup:271:                                                                  udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog.bkup:273:                                                                  udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null |  partimage -f3 -b restore $win7rec stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog.bkup:274:                                                                  udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | partimage -f3 -b restore $win7sys stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog.bkup:278:                                                              udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | partimage -f3 -b restore ${part} stdin
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog.bkup:381:                                                                  udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | partimage -f3 -b restore $part stdin;
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog.bkup:469:                                                                  udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null |  partimage -f3 -b restore $part stdin ;
/root/svn/0.32/trunk/src/buildroot/package/customize/fog/scripts/bin/fog.bkup:519:                                                      udp-receiver --nokbd --portbase ${port} --mcast-rdv-address ${storageip} 2>/dev/null | gunzip -f -c 2>/dev/null | dd of=${hd}

kenneth.sisco · Jan 14, 2016, 4:26 PM

We’re still having trouble narrowing this down, but is it safe to assume that if Multicast is working across the same VLAN that it’s no longer a fog issue?

Sebastian Roth · Jan 14, 2016, 5:09 PM

@kenneth.sisco said:

We’re still having trouble narrowing this down, but is it safe to assume that if Multicast is working across the same VLAN that it’s no longer a fog issue?

I would think so! Looked into igmp settings on your switches yet?

kenneth.sisco · Jan 14, 2016, 7:21 PM

@Sebastian-Roth Yes, we’ve looked at IGMP, but our topology is fairly complex. Currently the network team is working on moving rendezvous points closer to the server/clients and investigating potential issues in NXOS, as we recently migrated from Cisco IOS cores to Nexus.

Unfortunately, my visibility on this is very limited, so I’m doing my best to keep this rolling on the server side as we troubleshoot.

I’ll update with more info later as we progress.

kenneth.sisco · Jan 14, 2016, 9:23 PM

Cool - issue resolved!

This was absolutely an issue on our network, but we’re not sure if it was an old static Rendezvous Point that was configured on our core or if it was our VPC redundancy between our two nexus switches, but we were able to solve it by disabling pim on one of the nexuses and removing that static RP.

Thanks for the help on investigating, everyone. It’s much appreciated.

Multicasting Issues

226

12.1k

17.3k

155.3k