• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

FOG Multicast issue - stuck on starting to restore image - version 7547

Scheduled Pinned Locked Moved Solved
FOG Problems
4
57
20.0k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • A
    arainero @Tom Elliott
    last edited by arainero Jul 4, 2016, 12:47 PM Jul 4, 2016, 6:45 PM

    @Tom-Elliott @Wayne-Workman

    Sorry for the delay, here are the exact steps:

    For testing 1 client:

    1. Boot one computer into debug
    2. Enter udp-sender --file /opt/fog/.fogsettings --log /opt/fog/log/multicast.log --ttl 1 --nopointopoint into server and press enter. I now see
    [root@fogserv ~]# udp-sender --file /opt/fog/.fogsettings --log /opt/fog/log/multicast.log  --ttl 1 --nopointopoint
    Udp-sender 20120424
    Using full duplex mode
    
    
    1. Go to client computer and type this udp-receiver. On the client this immediately shows
    Udp-receiver 20120424
    UDP receiver for (stdout) at 192.168.1.118 on eth0
    received message, cap=00000009
    Connected as #0 to 192.168.1.3
    Listening to multicast on 232.168.1.3
    Press any key to start receiving data!
    
    

    On the server this immediately shows

    New connection from 192.168.1.118  (#0) 00000009
    Ready. Press any key to start sending data.
    
    1. I press enter on the server. Nothing happens.
    2. I now press enter on the client. I get
    Sending go signal
    

    Nothing happens on the server. I now press enter on the server and still nothing happens.

    I CTRL-C out on the client and see [root@fogserv ~]# re-xmits=0000000 ( 0.0%) slice=0112 - 0

    appear on the server. I CTRL-C on server and I am back to the prompt.

    If I repeat the steps to step 4 and hit enter on the client first instead of the server I get the same results.

    If I do the above steps again but use

    udp-sender --file /opt/fog/.fogsettings --log /opt/fog/log/multicast.log --ttl 1 --nopointopoint --mcast-data-address 192.168.1.255
    
    

    instead of the wiki command and hit enter on the client first I see the fog settings file displayed on the client. If I repeat that and hit enter on the server first I see the settings displayed. This also works with a 2nd client computer and does not matter if I hit enter on the client or server first.

    The --mcast-data-address 192.168.1.255 seems to be the magic argument that makes it “work” for the test scenario.

    Was this enough to determine anything or would you want me to do anything else to help narrow it down? If anyone wants to take a look through teamviewer or what-have-you I am more than open.

    Thanks again everyone and enjoy the 4th!

    1 Reply Last reply Reply Quote 0
    • A
      arainero
      last edited by Jul 7, 2016, 10:52 PM

      Should I update to trunk and do the same tests or hold off on that for now?

      1 Reply Last reply Reply Quote 0
      • S
        Sebastian Roth Moderator
        last edited by Jul 8, 2016, 9:16 AM

        @arainero What about using the options you see in the logs? cat /path/to/file | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 1 --max-wait 600 --portbase 65102 --full-duplex --ttl 32 --nokbd --nopointopoint

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        A 1 Reply Last reply Jul 8, 2016, 5:20 PM Reply Quote 0
        • A
          arainero @Sebastian Roth
          last edited by Jul 8, 2016, 5:20 PM

          @Sebastian-Roth

          If I use this modifed version of that line from the log it works.

          cat /opt/fog/.fogsettings | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 1 --max-wait 600 --full-duplex --nokbd --nopointopoint --mcast-data-address 192.168.1.255
          

          I had to remove portbase and the ttl flags and add the --mcast-data-address 192.168.1.255 flag. After doing that I was able to receive the fog settings files.

          I tried the command with just removing the portbase and ttl flags and not adding --mcast-data-address 192.168.1.255, but that didnt work.

          1 Reply Last reply Reply Quote 0
          • S
            Sebastian Roth Moderator
            last edited by Sebastian Roth Jul 8, 2016, 11:43 AM Jul 8, 2016, 5:38 PM

            @arainero Why did you have to remove --ttl and --portbase?? I still think something is wrong with the network setup. From the udp-cast man page I get that it would default to use 232.168.1.x (where x would be the last byte of the server IP address). Possibly your switch is layer 3 switch and does not allow those “out of range” IPs?

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            A 1 Reply Last reply Jul 8, 2016, 6:28 PM Reply Quote 1
            • A
              arainero @Sebastian Roth
              last edited by Jul 8, 2016, 6:28 PM

              @Sebastian-Roth I removed those to test to see if different scenarios would work. I tested with --mcast-data-address 232.168.1.255, 232.168.1.1, 232.168.1.0, etc, and those worked too.

              I actually added --ttl 32 and that worked. It’s just when I add --portbase 65102 it stops working. I tried a few different port variations with no luck.

              The switches in the letwork are layer 2. Multicast used to work fine and there have been no changes to the network. I will double check the switch configurations again though.

              W 2 Replies Last reply Jul 8, 2016, 9:23 PM Reply Quote 0
              • W
                Wayne Workman @arainero
                last edited by Wayne Workman Jul 9, 2016, 6:26 AM Jul 8, 2016, 9:23 PM

                @arainero are you sure fog firewall is off? Some places configure special ports for use with multicast (like mine).

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                Daily Clean Installation Results:
                https://fogtesting.fogproject.us/
                FOG Reporting:
                https://fog-external-reporting-results.fogproject.us/

                A 1 Reply Last reply Jul 9, 2016, 5:04 PM Reply Quote 0
                • W
                  Wayne Workman @arainero
                  last edited by Wayne Workman Jul 9, 2016, 6:28 AM Jul 9, 2016, 12:21 PM

                  @arainero said in FOG Multicast issue - stuck on starting to restore image - version 7547:

                  It’s just when I add --portbase 65102 it stops working. I tried a few different port variations with no luck.

                  Working off of this finding - I may have a fix for you.

                  Open up this file on your master node:
                  /var/www/html/fog/lib/service/multicasttask.class.php

                  Comment out line 91, do that by putting two forward slashes at the beginning of the line, like this:

                  0_1468066820472_alter multicast.png

                  Then restart the FOGMulticastManager like this:
                  service FOGMulticastManager restart

                  And then try multicast again. With any luck, it’ll work.

                  Even if it DOES work - we need to figure out what it is about your network that’s preventing you from defining a port, I hope you understand.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                  Daily Clean Installation Results:
                  https://fogtesting.fogproject.us/
                  FOG Reporting:
                  https://fog-external-reporting-results.fogproject.us/

                  A 1 Reply Last reply Jul 9, 2016, 5:20 PM Reply Quote 0
                  • S
                    Sebastian Roth Moderator
                    last edited by Jul 9, 2016, 4:23 PM

                    @arainero said:

                    Multicast used to work fine and there have been no changes to the network.

                    Can’t remember how many times I’ve heard this. But those kind of things usually don’t just stop working from one day to the other. Maybe it wasn’t you who changed anything. I really hope we can find out what’s wrong here and find a solution to it. Keeping my fingers crossed. Are you absolutely sure there are only layer 2 switches between the FOG server and the clients?

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    A 1 Reply Last reply Jul 9, 2016, 5:25 PM Reply Quote 0
                    • A
                      arainero @Wayne Workman
                      last edited by Jul 9, 2016, 5:04 PM

                      @Wayne-Workman

                      iptables and selinux are off. Firewalld is not installed either. Is there a firewall somewhere else that I have missed?

                      1 Reply Last reply Reply Quote 0
                      • A
                        arainero @Wayne Workman
                        last edited by arainero Jul 9, 2016, 11:27 AM Jul 9, 2016, 5:20 PM

                        @Wayne-Workman I tried this with no luck, but I expected it to work as you did. I had some interesting observations.

                        The new command I found in the multicast log was

                        cat /images/6_16_16/d1p1.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 2 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/6_16_16/d1p2.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 2 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/6_16_16/d2p1.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 2 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint;
                        

                        As expected there were no ports involved, but the multicast still didn’t work. However, if I took the first part of it and replaced it with the fogsettings file like in the wiki test it works. (This test was the test from the wiki and not a GUI multicast)

                        cat /opt/fog/.fogsettings | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 2 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint
                        

                        The only difference between the two commands is the file and the fact that I am running the second one manually through the terminal and receiving it manually with udp-receiver.

                        Is there a different process when doing this test manually compared to when it is done through the GUI?

                        Here is the multicast log

                        [07-09-16 1:08:48 pm] | Task (1) Multi-Cast Task has started.
                        [07-09-16 1:08:48 pm] | CMD: cat /images/6_16_16/d1p1.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 1 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/6_16_16/d1p2.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 1 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/6_16_16/d2p1.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 1 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint;
                        [07-09-16 1:08:48 pm] | Task (1) Multi-Cast Task sending on base port: 53324
                        [07-09-16 1:08:48 pm] | Task (1) 1 client(s) found.
                        Broadcasting control to 224.0.0.1
                        UDP sender for (stdin) at 192.168.1.3 on eth0 
                        [07-09-16 1:08:48 pm] | Task (1) /images/6_16_16 image file found.
                        Udp-sender 20120424
                        [07-09-16 1:08:48 pm] | Task (1) Multi-Cast Task has been cleaned.
                        [07-09-16 1:08:48 pm] | Task (1) Multi-Cast Task is new!
                        [07-09-16 1:08:48 pm] | 1 task found
                        [07-09-16 1:08:48 pm] | 0 tasks to be cleaned
                        [07-09-16 1:08:38 pm] | Sleeping for 10 seconds to ensure tasks are properly submitted
                        [07-09-16 1:08:28 pm] * No tasks found!
                        [07-09-16 1:08:18 pm] * No tasks found!
                        [07-09-16 1:08:18 pm] * Starting service loop
                        [07-09-16 1:08:18 pm] * Checking for new items every 10 seconds
                        [07-09-16 1:08:18 pm] * Starting MulticastManager Service
                        
                        1 Reply Last reply Reply Quote 0
                        • A
                          arainero @Sebastian Roth
                          last edited by Jul 9, 2016, 5:25 PM

                          @Sebastian-Roth I know, I turned into one of those people. But as far as I can tell nothing on the network has changed. It is a very basic setup. Internet comes in, hits the pfsense router which goes to a layer 2 meraki switch (which is trunked to another identical one) and out to the computers.

                          The only other device on the network is a wireless AP and a few gaming consoles. Other than that every computer has a direct ethernet cord to it. All of these devices were here before when multicast was working.

                          It truly sounds network related somehow, but I have no idea what could be causing it at this point.

                          On Monday I plan to hook the router (for the pxe boot routing), 2 pc’s for multicasting, and an admin computer (for the web GUI access) to a dumb switch and see what happens when I cut everything else out. That should rule out any network issues, right?

                          W 1 Reply Last reply Jul 9, 2016, 5:43 PM Reply Quote 0
                          • W
                            Wayne Workman @arainero
                            last edited by Wayne Workman Jul 9, 2016, 11:45 AM Jul 9, 2016, 5:43 PM

                            @arainero wait a second there, the multicast is going through the router? That could be the problem, it’s probably blocking the needed port and/or UDP traffic.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                            Daily Clean Installation Results:
                            https://fogtesting.fogproject.us/
                            FOG Reporting:
                            https://fog-external-reporting-results.fogproject.us/

                            A 1 Reply Last reply Jul 9, 2016, 7:17 PM Reply Quote 0
                            • A
                              arainero @Wayne Workman
                              last edited by Jul 9, 2016, 7:17 PM

                              @Wayne-Workman The multicast doesn’t go through the router, it is just the edge device and I was just giving a brief rundown of the network and how basic it was.

                              As far as I know the router can’t block interlan traffic. Also, no changes have been made to the router either (I know how much you guys love hearing that).

                              W 1 Reply Last reply Jul 9, 2016, 10:17 PM Reply Quote 0
                              • W
                                Wayne Workman @arainero
                                last edited by Wayne Workman Jul 9, 2016, 4:17 PM Jul 9, 2016, 10:17 PM

                                @arainero

                                What side of the router is the fog server on? You didn’t include that part, you just said three computers on a dumb switch. Where is the fog server?

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                Daily Clean Installation Results:
                                https://fogtesting.fogproject.us/
                                FOG Reporting:
                                https://fog-external-reporting-results.fogproject.us/

                                A 1 Reply Last reply Jul 10, 2016, 4:29 PM Reply Quote 0
                                • S
                                  Sebastian Roth Moderator
                                  last edited by Sebastian Roth Jul 10, 2016, 3:57 AM Jul 10, 2016, 9:55 AM

                                  @arainero Thanks for the new logs, tests and explanations. I am pretty sure we are headed the right way with that. Doing the test with a dump switch and just the FOG server and two clients on it is definitely a good idea - although your other setup also sounds quite simple. Still give it a try with that dump switch I’d say. A couple of things that got my attention:

                                  • The timestamps in the last log output you posted are in reverse?!? That does not make any sense to me.
                                  • two meraki switches being trunked (was this setup always like that? which trunking protocol is being used?) - I’ve never done multicasting over a trunked link
                                  • have you ever checked on your FOG server if those udp-sender processes are actually started? After checking the log also run ps ax | grep udp
                                  • AFAIK there should be no difference between running the command by hand and it being run by FOG when scheduling a task. But please give the following steps a try to see if it makes any difference. Schedule a multicast task for two of your clients plus unicast debug deploy tasks for another two clients. Make sure the multicast clients do not boot up (disconnect from the LAN so they don’t wake up) and start the two unicast clients. When they are up run udp-receiver --nokbd --ttl 32 --mcast-rdv-address 192.168.1.3 | gunzip > /dev/sda1

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  W A 3 Replies Last reply Jul 10, 2016, 4:18 PM Reply Quote 0
                                  • W
                                    Wayne Workman @Sebastian Roth
                                    last edited by Jul 10, 2016, 4:18 PM

                                    @Sebastian-Roth said in FOG Multicast issue - stuck on starting to restore image - version 7547:

                                    The timestamps in the last log output you posted are in reverse?!?

                                    In Fog’s web interface, there’s a check box to reverse the logs, that way the newest ones appear at the top. I feel it should be default, because with a large log you can’t always get to the bottom of the file by using the web interface to show 1000 lines.

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                    Daily Clean Installation Results:
                                    https://fogtesting.fogproject.us/
                                    FOG Reporting:
                                    https://fog-external-reporting-results.fogproject.us/

                                    1 Reply Last reply Reply Quote 0
                                    • A
                                      arainero @Sebastian Roth
                                      last edited by Jul 10, 2016, 4:27 PM

                                      @Sebastian-Roth

                                      Could the time stamps be like that since I had the log sorted from top down?

                                      Trunking might have been the wrong term. They are simply connected to each other through SFP ports with twinax cable. Like plugging a switch into another switch with ethernet.

                                      The udp sender appears to be in working order. This is what I get after starting a multicast session:

                                      [root@fogserv ~]# ps ax | grep udp
                                      28869 ?        S      0:00 sh -c cat /images/6_16_16/d1p1.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 2 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/6_16_16/d1p2.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 2 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint;cat /images/6_16_16/d2p1.img | /usr/local/sbin/udp-sender --interface eth0 --min-receivers 2 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint;
                                      28871 ?        S      0:00 /usr/local/sbin/udp-sender --interface eth0 --min-receivers 2 --max-wait 180 --mcast-data-address 232.168.1.0 --full-duplex --ttl 32 --nokbd --nopointopoint
                                      28881 pts/1    S+     0:00 grep udp
                                      

                                      Doing the debug test seems to have worked.
                                      alt text

                                      One thing I noticed that might be odd is that when I went to the task management page and to active multicast tasks it said 4 hosts. There are only 2 members in the group. Is this indicative of anything wrong?

                                      alt text

                                      T 1 Reply Last reply Jul 11, 2016, 9:55 AM Reply Quote 0
                                      • A
                                        arainero @Wayne Workman
                                        last edited by Jul 10, 2016, 4:29 PM

                                        @Wayne-Workman The fog server is on the LAN along with the client computers. The router is the edge device and handles the DHCP for the network.

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          Sebastian Roth Moderator
                                          last edited by Jul 11, 2016, 9:45 AM

                                          @arainero Do I get this right? You did as I suggested: started a multicast using the web gui and started the multicast receive on two clients by hand. That worked!? So why wouldn’t it work when the other two clients run into the multicast session???

                                          Could you please do this same test again but remove the --mcast-data-address 232.168.1.0 setting. This should not be necessary I reckon. After that give it another try with a plain multicast session. But first make sure that you don’t have any old udp-sender processes running on the FOG server (ps ax | grep udp and killall udp-sender in case there are orphaned processes around) and clear the (multicast) task lists in the web gui!

                                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                          1 Reply Last reply Reply Quote 0
                                          • 1
                                          • 2
                                          • 3
                                          • 2 / 3
                                          2 / 3
                                          • First post
                                            39/57
                                            Last post

                                          151

                                          Online

                                          12.1k

                                          Users

                                          17.3k

                                          Topics

                                          155.4k

                                          Posts
                                          Copyright © 2012-2024 FOG Project