Multicast failing to start, hangs on partclone starting to restore image



  • Hello,

    I have been having this problem for months now with multicast. I am running the latest svn release 2949. I have downgraded, upgraded, and sidegraded to no avail. I have gotten the multicasting to work before but the next time I need to use it (a few weeks later) it stops working again. I have to spend at least 4-7 hours playing with FOG by uninstalling / reinstalling to get it to work again.

    I have followed directions here several times to no avail [url]http://www.fogproject.org/wiki/index.php/Troubleshooting_a_multicast[/url] . I have done the tests and every computer can manually join a session.

    My fog multicast log looks like this:

    [CODE][01-30-15 1:01:33 am] | Task (8) is new!
    [01-30-15 1:01:33 am] | Task (8) /images/12817image image file found.
    [01-30-15 1:01:33 am] | Task (8) 6 client(s) found.
    [01-30-15 1:01:33 am] | Task (8) sending on base port: 60118
    [01-30-15 1:01:33 am] CMD:
    [01-30-15 1:01:33 am] | Task (8) has started.
    [01-30-15 1:01:43 am] | Task (8) is no longer running.
    [01-30-15 1:01:43 am] | Task (8) has been cleaned as complete.
    [01-30-15 1:01:53 am] * No Tasks Found!
    [01-30-15 1:02:03 am] * No Tasks Found!
    [01-30-15 1:02:13 am] * No Tasks Found!
    [01-30-15 1:02:23 am] * No Tasks Found!
    [01-30-15 1:02:33 am] * No Tasks Found!
    [01-30-15 1:02:43 am] * No Tasks Found!
    [01-30-15 1:02:53 am] * No Tasks Found!
    [01-30-15 1:03:03 am] * No Tasks Found!
    [01-30-15 1:03:13 am] * No Tasks Found!
    [01-30-15 1:03:23 am] * No Tasks Found!
    [/CODE]

    Messages Log
    [CODE]
    Jan 29 19:59:45 fogserv kernel: NFSD: starting 90-second grace period
    Jan 29 20:01:00 fogserv xinetd[7462]: START: tftp pid=9598 from=192.168.1.142
    Jan 29 20:01:00 fogserv in.tftpd[9599]: tftp: client does not accept options
    Jan 29 20:01:00 fogserv in.tftpd[9601]: tftp: client does not accept options
    Jan 29 20:01:00 fogserv in.tftpd[9603]: tftp: client does not accept options
    Jan 29 20:01:00 fogserv in.tftpd[9605]: tftp: client does not accept options
    Jan 29 20:01:00 fogserv in.tftpd[9607]: tftp: client does not accept options
    Jan 29 20:01:04 fogserv in.tftpd[9626]: tftp: client does not accept options
    Jan 29 20:01:30 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.142:972 for /images (/images)
    Jan 29 20:01:31 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.139:732 for /images (/images)
    Jan 29 20:01:31 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.138:846 for /images (/images)
    Jan 29 20:01:32 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.141:721 for /images (/images)
    Jan 29 20:01:32 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.140:740 for /images (/images)
    Jan 29 20:01:35 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.137:847 for /images (/images)
    Jan 29 20:01:37 fogserv in.tftpd[9630]: tftpd: read: Connection refused
    Jan 29 20:01:37 fogserv in.tftpd[9629]: tftpd: read: Connection refused
    [/CODE]

    It seems when a task is created it “completes” it almost instantly. I believe the problem is here.

    I have tried restarting the computer, services, and etc. I am running Centos 6.

    Thank you for the help.



  • The changes Tom made corrected the issue. Thanks again!


  • Developer

    [quote=“arainero, post: 41526, member: 27492”]
    Test 2 worked as well. I want to point out that on the wiki it says “udb-receiver” where i imagine it’s supposed to be “udp-receiver”.
    [/quote]

    wiki has been fixed thanks for reporting!


  • Senior Developer

    SVN 2951 should fix this issue for you.



  • [quote=“Uncle Frank, post: 41525, member: 28116”]I am still very new to this project and don’t have a deep insight into the PHP code yet. But taking a look at those lines where these warnings occoured it seams like the image files are missing or can’t be read/found!? Where are your images? What do you see running this command:
    [CODE]ls -al /images/*[/CODE][/quote]

    I get:

    [CODE]/images/11515:
    total 620975000
    drwxrwxrwx 2 root root 4096 Jan 16 00:32 .
    drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …
    -rwxrwxrwx 1 root root 512 Jan 16 00:11 d1.mbr
    -rwxrwxrwx 1 root root 0 Jan 16 00:11 d1.original.swapuuids
    -rwxrwxrwx 1 root root 8522347 Jan 16 00:11 d1p1.img
    -rwxrwxrwx 1 root root 23625716517 Jan 16 00:32 d1p2.img
    -rwxrwxrwx 1 root root 512 Jan 16 00:32 d2.mbr
    -rwxrwxrwx 1 root root 0 Jan 16 00:32 d2.original.swapuuids
    -rwxrwxrwx 1 root root 612244074192 Jan 16 05:35 d2p1.img

    /images/12815image:
    total 8
    drwxrwxrwx 2 root root 4096 Jan 29 18:17 .
    drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …

    /images/12817image:
    total 624316568
    drwxrwxrwx 2 root root 4096 Jan 29 18:51 .
    drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …
    -rwxrwxrwx 1 root root 512 Jan 28 21:49 d1.mbr
    -rwxrwxrwx 1 root root 8520325 Jan 28 21:49 d1p1.img
    -rwxrwxrwx 1 root root 24696905617 Jan 28 22:09 d1p2.img
    -rwxrwxrwx 1 root root 512 Jan 28 22:09 d2.mbr
    -rwxrwxrwx 1 root root 614594652526 Jan 29 03:17 d2p1.img

    /images/dev:
    total 16
    drwxrwxrwx. 4 root root 4096 Jan 29 18:15 .
    drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …
    drwxrwxrwx 2 root root 4096 Jan 16 14:05 fcaa14310adb
    drwxrwxrwx 2 root root 4096 Jan 29 18:15 fcaa14310c42
    -rwxrwxrwx. 1 root root 0 Jan 15 22:15 .mntcheck

    /images/postdownloadscripts:
    total 12
    drwxrwxrwx. 2 root root 4096 Jan 15 22:15 .
    drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …
    -rwxrwxrwx. 1 root root 233 Jan 15 22:15 fog.postdownload
    [/CODE]

    In the process of me removing FOG and reinstalling it in attempts to get it to work I followed the following from the wiki: [url]http://www.fogproject.org/wiki/index.php/Uninstall_FOG[/url]. EXCEPT deleting the images folder. I noticed that when I reinstalled FOG that my images didn’t show up until I created a “new” image and then I moved the old files into the newly created folder. I believe a database entry had to be made for them to show up.

    I’d like to mention that I am able to do a Unicast just fine, so I don’t believe there is a problem reading the image. Maybe multicast reads it differently though?



  • I have done both tests before with no problems, but I will do them again.

    I had a successful test 1:

    [CODE][root@fogserv bin]# udp-sender --file /opt/fog/.fogsettings --log /opt/fog/log/multicast.log --ttl 1 --nopointopoint
    Udp-sender 20120424
    Using full duplex mode
    New connection from 192.168.1.142 (#0) 00000009
    Ready. Press any key to start sending data.
    [/CODE]
    [IMG]http://i.imgur.com/pTsIG6t.jpg[/IMG]

    Test 2 worked as well. I want to point out that on the wiki it says “udb-receiver” where i imagine it’s supposed to be “udp-receiver”.

    [CODE][root@fogserv bin]# udp-sender --file /opt/fog/.fogsettings --log /opt/fog/log/multicast.log --ttl 32 --nopointopoint
    Udp-sender 20120424
    Using full duplex mode
    New connection from 192.168.1.141 (#0) 00000009
    Ready. Press any key to start sending data.
    New connection from 192.168.1.142 (#1) 00000009
    Ready. Press any key to start sending data.
    [root@fogserv bin]# re-xmits=0000000 ( 0.0%) slice=0112 - 0
    [/CODE]

    Here is what was added to the multicast log:

    [CODE]11:21:40.186529 Using mcast address 232.168.1.3
    11:21:40.186615 UDP sender for /opt/fog/.fogsettings at 192.168.1.3 on eth0
    11:21:40.186631 Broadcasting control to 192.168.1.255
    [01-30-15 4:21:40 pm] * No Tasks Found!
    [01-30-15 4:21:50 pm] * No Tasks Found!
    [01-30-15 4:22:00 pm] * No Tasks Found!
    [01-30-15 4:22:10 pm] * No Tasks Found!
    [01-30-15 4:22:20 pm] * No Tasks Found!
    [01-30-15 4:22:30 pm] * No Tasks Found!
    11:22:37.110626 Starting transfer: 00000009
    11:22:37.123028 Transfer complete.
    11:22:37.123072 Disconnecting #0 (192.168.1.142)
    11:22:37.123107
    [01-30-15 4:22:40 pm] * No Tasks Found!
    [01-30-15 4:22:50 pm] * No Tasks Found!
    [01-30-15 4:23:00 pm] * No Tasks Found!
    [01-30-15 4:23:10 pm] * No Tasks Found!
    [01-30-15 4:23:20 pm] * No Tasks Found!
    [01-30-15 4:23:30 pm] * No Tasks Found!
    [01-30-15 4:23:40 pm] * No Tasks Found!
    [01-30-15 4:23:50 pm] * No Tasks Found!
    [01-30-15 4:24:00 pm] * No Tasks Found!
    [01-30-15 4:24:11 pm] * No Tasks Found!
    [01-30-15 4:24:21 pm] * No Tasks Found!
    [01-30-15 4:24:31 pm] * No Tasks Found!
    [01-30-15 4:24:41 pm] * No Tasks Found!
    [01-30-15 4:24:51 pm] * No Tasks Found!
    [01-30-15 4:25:01 pm] * No Tasks Found!
    [01-30-15 4:25:11 pm] * No Tasks Found!
    [01-30-15 4:25:21 pm] * No Tasks Found!
    [01-30-15 4:25:31 pm] * No Tasks Found!
    [01-30-15 4:25:41 pm] * No Tasks Found!
    [01-30-15 4:25:51 pm] * No Tasks Found!
    [01-30-15 4:26:01 pm] * No Tasks Found!
    [01-30-15 4:26:11 pm] * No Tasks Found!
    [01-30-15 4:26:21 pm] * No Tasks Found!
    [01-30-15 4:26:31 pm] * No Tasks Found!
    11:26:34.969640 Using mcast address 232.168.1.3
    11:26:34.969731 UDP sender for /opt/fog/.fogsettings at 192.168.1.3 on eth0
    11:26:34.969747 Broadcasting control to 224.0.0.1
    [01-30-15 4:26:41 pm] * No Tasks Found!
    [01-30-15 4:26:51 pm] * No Tasks Found!
    [01-30-15 4:27:01 pm] * No Tasks Found!
    [01-30-15 4:27:11 pm] * No Tasks Found!
    [01-30-15 4:27:21 pm] * No Tasks Found!
    [01-30-15 4:27:31 pm] * No Tasks Found!
    [01-30-15 4:27:41 pm] * No Tasks Found!
    [01-30-15 4:27:51 pm] * No Tasks Found!
    [01-30-15 4:28:01 pm] * No Tasks Found!
    11:28:03.473599 Starting transfer: 00000009
    11:28:03.475066 Transfer complete.
    11:28:03.475107 Disconnecting #0 (192.168.1.141)
    11:28:03.475134 Disconnecting #1 (192.168.1.142)
    11:28:03.475162
    [01-30-15 4:28:11 pm] * No Tasks Found![/CODE]

    I would also like to add that we have a pfSense router and two Cisco Meraki switches. I do not believe they are causing problems since multicast has worked before, but I thought you should know.


  • Developer

    [quote=“arainero, post: 41506, member: 27492”]I also just noticed this populating in the bash prompt:

    [CODE][root@fogserv bin]#
    PHP Warning: in_array() expects parameter 2 to be array, null given in /var/www/html/fog/lib/fog/MulticastManager.class.php on line 16
    PHP Warning: natsort() expects parameter 1 to be array, null given in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 152
    PHP Warning: Invalid argument supplied for foreach() in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 153
    PHP Warning: implode(): Argument must be an array in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 155
    PHP Warning: in_array() expects parameter 2 to be array, null given in /var/www/html/fog/lib/fog/MulticastManager.class.php on line 16
    PHP Warning: natsort() expects parameter 1 to be array, null given in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 152
    PHP Warning: Invalid argument supplied for foreach() in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 153
    PHP Warning: implode(): Argument must be an array in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 155
    [/CODE][/quote]

    I am still very new to this project and don’t have a deep insight into the PHP code yet. But taking a look at those lines where these warnings occoured it seams like the image files are missing or can’t be read/found!? Where are your images? What do you see running this command:
    [CODE]ls -al /images/*[/CODE]


  • Developer

    [LIST]
    []What are your results when testing the 1 client troubleshooting guide?[url]http://www.fogproject.org/wiki/index.php/Troubleshooting_a_multicast#Testing_1_Client[/url]
    [
    ]What are your results when testing the 2 client troubleshooting guide? [url]http://www.fogproject.org/wiki/index.php/Troubleshooting_a_multicast#Testing_2_Clients[/url]
    [/LIST]
    The results here really do matter in narrowing down the issue.



  • Looks like everything is running:

    [CODE][root@fogserv ~]# ps aux | grep FOG
    root 983 0.0 0.0 103252 824 pts/1 S+ 22:27 0:00 grep FOG
    root 9100 0.0 0.1 251496 11216 pts/0 S 19:59 0:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FOGImageReplicator
    root 9146 0.0 0.1 252008 11800 pts/0 S 19:59 0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
    root 9181 0.0 0.1 251240 11028 pts/0 S 19:59 0:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
    root 10669 0.0 0.1 251756 11692 pts/0 S 20:17 0:02 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
    [/CODE]


  • Developer

    Are all FOG services on the server running??
    [CODE]ps aux | grep FOG[/CODE]

    If not, restart all of them (sometimes you even need to restart them twice):
    [CODE]service FOGImageReplicator restart
    service FOGScheduler restart
    service FOGMulticastManager restart[/CODE]



  • FOG Install Log

    [CODE]CentOS

      ..#######:.    ..,#,..    .::##::. 
    

    .:###### .:;####:…;#;…
    …##… …##;,;##::::.##…
    ,# …##…##:::## …::
    ## .::###,##. . ##.::#.:######::.
    …##:::###::…#. … .#…#. #…#:::.
    …:####:… …##…##::## … #
    # . …##:,;##;:::#: … ##…
    .# . .:;####;::::.##:::;#:…
    # …:;###…

    ###########################################

    FOG

    Free Computer Imaging Solution

    http://www.fogproject.org/

    Developers:

    Chuck Syperski

    Jian Zhang

    Peter Gilchrist

    Tom Elliott

    GNU GPL Version 3

    ###########################################

    Version: 1.3.0 Installer/Updater

    • Found FOG Settings from previous install at: /opt/fog/.fogsettings
    • Performing upgrade using these settings…

    Staring Redhat / CentOS Installation.

    #####################################################################

    FOG now has everything it needs to setup your server, but please
    understand that this script will overwrite any setting you may
    have setup for services like DHCP, apache, pxe, tftp, and NFS.

    It is not recommended that you install this on a production system
    as this script modifies many of your system settings.

    This script should be run by the root user on Redhat or with sudo on Ubuntu.

    ** Notice ** Redhat users will need to disable SELinux and iptables in
    order to use FOG
    Please see our wiki for more information at http://www.fogproject.org/wiki

    Here are the settings FOG will use:
    Base Linux: Redhat
    Detected Linux Distribution: CentOS
    Installation Type: Normal Server
    Server IP Address: 192.168.1.3
    DHCP router Address: 192.168.1.1
    DHCP DNS Address: 192.168.1.1
    Interface: eth0
    Using FOG DHCP: 0
    Internationalization: 0
    Image Storage Location:
    Donate: 0

        DHCP will NOT be setup but you must setup your
        current DHCP server to use FOG for PXE services.
    
        On a Linux DHCP server you must set:
            next-server
    
        On a Windows DHCP server you must set:
            option 066 & 067
    
          Option 066 is the IP of the FOG Server: (e.g. 192.168.1.3)
          Option 067 is the undionly.kpxe file: (e.g. undionly.kpxe)
    

    Are you sure you wish to continue (Y/N)
    Installation Started…

    Installing required packages, if this fails
    make sure you have an active internet connection.

    • Packages to be installed: httpd php php-cli php-common php-gd php-mysql mysql mysql-server tftp-server nfs-utils vsftpd net-tools wget xinetd tar gzip make m4 gcc gcc-c++ lftp

    • Skipping package: httpd (Already installed)

    • Skipping package: php (Already installed)

    • Skipping package: php-cli (Already installed)

    • Skipping package: php-common (Already installed)

    • Skipping package: php-gd (Already installed)

    • Skipping package: php-mysql (Already installed)

    • Skipping package: mysql (Already installed)

    • Skipping package: mysql-server (Already installed)

    • Skipping package: tftp-server (Already installed)

    • Skipping package: nfs-utils (Already installed)

    • Skipping package: vsftpd (Already installed)

    • Skipping package: net-tools (Already installed)

    • Skipping package: wget (Already installed)

    • Skipping package: xinetd (Already installed)

    • Skipping package: tar (Already installed)

    • Skipping package: gzip (Already installed)

    • Skipping package: make (Already installed)

    • Skipping package: m4 (Already installed)

    • Skipping package: gcc (Already installed)

    • Skipping package: gcc-c++ (Already installed)

    • Skipping package: lftp (Already installed)

    Confirming package installation.

    • Checking package: httpd…OK
    • Checking package: php…OK
    • Checking package: php-cli…OK
    • Checking package: php-common…OK
    • Checking package: php-gd…OK
    • Checking package: php-mysql…OK
    • Checking package: mysql…OK
    • Checking package: mysql-server…OK
    • Checking package: tftp-server…OK
    • Checking package: nfs-utils…OK
    • Checking package: vsftpd…OK
    • Checking package: net-tools…OK
    • Checking package: wget…OK
    • Checking package: xinetd…OK
    • Checking package: tar…OK
    • Checking package: gzip…OK
    • Checking package: make…OK
    • Checking package: m4…OK
    • Checking package: gcc…OK
    • Checking package: gcc-c++…OK
    • Checking package: lftp…OK

    Configuring services.

    What is the storage location for your images directory? (/images)  * Setting up and starting MySQL...OK
    
    • Backing up user reports…OK

    • Setting up and starting Apache Web Server…OK

    • Changing permissions on apache log files…OK

    • Downloading kernels and inits…OK

    • Copying back any custom hook files.

    • Copying back any custom report files.
      You still need to install/update your database schema.
      This can be done by opening a web browser and going to:

      http://192.168.1.3/fog/management

    Press [Enter] key when database is updated/installed.

    • Setting up storage…OK
    • Setting up and starting DHCP Server…Skipped
    • Setting up and starting TFTP and PXE Servers…OK
    • Setting up and starting VSFTP Server…OK
    • Setting up sudo settings…OK
    • Setting up FOG Snapins…OK
    • Setting up and building UDPCast…OK
    • Installing init scripts…OK
    • Setting up FOG Services…OK
    • Setting up FOG Utils…OK
    • Starting FOG Multicast Management Server…OK
    • Starting FOG Image Replicator Server…OK
    • Starting FOG Task Scheduler Server…OK
    • Starting FOG Snapin Replicator Server…OK
    • Configuring Fresh Clam…OK
    • Setting up and starting NFS Server…OK

    Setup complete!

    You can now login to the FOG Management Portal using
    the information listed below. The login information
    is only if this is the first install.

    This can be done by opening a web browser and going to:

      http://192.168.1.3/fog/management
    
      Default User:
            Username: fog
            Password: password
    

    [/CODE]



  • I also just noticed this populating in the bash prompt:

    [CODE][root@fogserv bin]#
    PHP Warning: in_array() expects parameter 2 to be array, null given in /var/www/html/fog/lib/fog/MulticastManager.class.php on line 16
    PHP Warning: natsort() expects parameter 1 to be array, null given in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 152
    PHP Warning: Invalid argument supplied for foreach() in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 153
    PHP Warning: implode(): Argument must be an array in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 155
    PHP Warning: in_array() expects parameter 2 to be array, null given in /var/www/html/fog/lib/fog/MulticastManager.class.php on line 16
    PHP Warning: natsort() expects parameter 1 to be array, null given in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 152
    PHP Warning: Invalid argument supplied for foreach() in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 153
    PHP Warning: implode(): Argument must be an array in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 155
    [/CODE]



  • [quote=“Tom Elliott, post: 41504, member: 7271”]Also as its specific to multicast, is the current method inupgrade/sidegrade setup? Can you truncate current multicastSessions and multicastSessionsAssoc tables?[/quote]

    I have done the truncate to those tables multiple times trying to fix this problem.

    I have selinux and iptables disabled.

    I am able to do a unicast with no problem. If you’d like remote access to the server I can arrange this.

    Thanks!


  • Senior Developer

    Also as its specific to multicast, is the current method inupgrade/sidegrade setup? Can you truncate current multicastSessions and multicastSessionsAssoc tables?


  • Senior Developer

    Have you disabled selinux and stopped iptables?


Log in to reply
 

446
Online

39174
Users

10821
Topics

102951
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.