Multicast failing to start, hangs on partclone starting to restore image

arainero · Jan 30, 2015, 1:12 AM

Hello,

I have been having this problem for months now with multicast. I am running the latest svn release 2949. I have downgraded, upgraded, and sidegraded to no avail. I have gotten the multicasting to work before but the next time I need to use it (a few weeks later) it stops working again. I have to spend at least 4-7 hours playing with FOG by uninstalling / reinstalling to get it to work again.

I have followed directions here several times to no avail [url]http://www.fogproject.org/wiki/index.php/Troubleshooting_a_multicast[/url] . I have done the tests and every computer can manually join a session.

My fog multicast log looks like this:

[CODE][01-30-15 1:01:33 am] | Task (8) is new!
[01-30-15 1:01:33 am] | Task (8) /images/12817image image file found.
[01-30-15 1:01:33 am] | Task (8) 6 client(s) found.
[01-30-15 1:01:33 am] | Task (8) sending on base port: 60118
[01-30-15 1:01:33 am] CMD:
[01-30-15 1:01:33 am] | Task (8) has started.
[01-30-15 1:01:43 am] | Task (8) is no longer running.
[01-30-15 1:01:43 am] | Task (8) has been cleaned as complete.
[01-30-15 1:01:53 am] * No Tasks Found!
[01-30-15 1:02:03 am] * No Tasks Found!
[01-30-15 1:02:13 am] * No Tasks Found!
[01-30-15 1:02:23 am] * No Tasks Found!
[01-30-15 1:02:33 am] * No Tasks Found!
[01-30-15 1:02:43 am] * No Tasks Found!
[01-30-15 1:02:53 am] * No Tasks Found!
[01-30-15 1:03:03 am] * No Tasks Found!
[01-30-15 1:03:13 am] * No Tasks Found!
[01-30-15 1:03:23 am] * No Tasks Found!
[/CODE]

Messages Log
[CODE]
Jan 29 19:59:45 fogserv kernel: NFSD: starting 90-second grace period
Jan 29 20:01:00 fogserv xinetd[7462]: START: tftp pid=9598 from=192.168.1.142
Jan 29 20:01:00 fogserv in.tftpd[9599]: tftp: client does not accept options
Jan 29 20:01:00 fogserv in.tftpd[9601]: tftp: client does not accept options
Jan 29 20:01:00 fogserv in.tftpd[9603]: tftp: client does not accept options
Jan 29 20:01:00 fogserv in.tftpd[9605]: tftp: client does not accept options
Jan 29 20:01:00 fogserv in.tftpd[9607]: tftp: client does not accept options
Jan 29 20:01:04 fogserv in.tftpd[9626]: tftp: client does not accept options
Jan 29 20:01:30 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.142:972 for /images (/images)
Jan 29 20:01:31 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.139:732 for /images (/images)
Jan 29 20:01:31 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.138:846 for /images (/images)
Jan 29 20:01:32 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.141:721 for /images (/images)
Jan 29 20:01:32 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.140:740 for /images (/images)
Jan 29 20:01:35 fogserv rpc.mountd[9341]: authenticated mount request from 192.168.1.137:847 for /images (/images)
Jan 29 20:01:37 fogserv in.tftpd[9630]: tftpd: read: Connection refused
Jan 29 20:01:37 fogserv in.tftpd[9629]: tftpd: read: Connection refused
[/CODE]

It seems when a task is created it “completes” it almost instantly. I believe the problem is here.

I have tried restarting the computer, services, and etc. I am running Centos 6.

Thank you for the help.

Tom Elliott · Jan 30, 2015, 1:18 AM

Have you disabled selinux and stopped iptables?

Tom Elliott · Jan 30, 2015, 1:21 AM

Also as its specific to multicast, is the current method inupgrade/sidegrade setup? Can you truncate current multicastSessions and multicastSessionsAssoc tables?

arainero · Jan 30, 2015, 1:27 AM

[quote=“Tom Elliott, post: 41504, member: 7271”]Also as its specific to multicast, is the current method inupgrade/sidegrade setup? Can you truncate current multicastSessions and multicastSessionsAssoc tables?[/quote]

I have done the truncate to those tables multiple times trying to fix this problem.

I have selinux and iptables disabled.

I am able to do a unicast with no problem. If you’d like remote access to the server I can arrange this.

Thanks!

arainero · Jan 30, 2015, 1:29 AM

I also just noticed this populating in the bash prompt:

[CODE][root@fogserv bin]#
PHP Warning: in_array() expects parameter 2 to be array, null given in /var/www/html/fog/lib/fog/MulticastManager.class.php on line 16
PHP Warning: natsort() expects parameter 1 to be array, null given in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 152
PHP Warning: Invalid argument supplied for foreach() in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 153
PHP Warning: implode(): Argument must be an array in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 155
PHP Warning: in_array() expects parameter 2 to be array, null given in /var/www/html/fog/lib/fog/MulticastManager.class.php on line 16
PHP Warning: natsort() expects parameter 1 to be array, null given in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 152
PHP Warning: Invalid argument supplied for foreach() in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 153
PHP Warning: implode(): Argument must be an array in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 155
[/CODE]

arainero · Jan 30, 2015, 1:35 AM

FOG Install Log

[CODE]CentOS

  ..#######:.    ..,#,..    .::##::.

.:###### .:;####:…;#;…
…##… …##;,;##::::.##…
,# …##…##:::## …::
## .::###,##. . ##.::#.:######::.
…##:::###::…#. … .#…#. #…#:::.
…:####:… …##…##::## … #
# . …##:,;##;:::#: … ##…
.# . .:;####;::::.##:::;#:…
# …:;###…

###########################################

FOG

Free Computer Imaging Solution

http://www.fogproject.org/

Developers:

Chuck Syperski

Jian Zhang

Peter Gilchrist

Tom Elliott

GNU GPL Version 3

###########################################

Version: 1.3.0 Installer/Updater

Found FOG Settings from previous install at: /opt/fog/.fogsettings
Performing upgrade using these settings…

Staring Redhat / CentOS Installation.

#####################################################################

FOG now has everything it needs to setup your server, but please
understand that this script will overwrite any setting you may
have setup for services like DHCP, apache, pxe, tftp, and NFS.

It is not recommended that you install this on a production system
as this script modifies many of your system settings.

This script should be run by the root user on Redhat or with sudo on Ubuntu.

** Notice ** Redhat users will need to disable SELinux and iptables in
order to use FOG
Please see our wiki for more information at http://www.fogproject.org/wiki

Here are the settings FOG will use:
Base Linux: Redhat
Detected Linux Distribution: CentOS
Installation Type: Normal Server
Server IP Address: 192.168.1.3
DHCP router Address: 192.168.1.1
DHCP DNS Address: 192.168.1.1
Interface: eth0
Using FOG DHCP: 0
Internationalization: 0
Image Storage Location:
Donate: 0

    DHCP will NOT be setup but you must setup your
    current DHCP server to use FOG for PXE services.

    On a Linux DHCP server you must set:
        next-server

    On a Windows DHCP server you must set:
        option 066 & 067

      Option 066 is the IP of the FOG Server: (e.g. 192.168.1.3)
      Option 067 is the undionly.kpxe file: (e.g. undionly.kpxe)

Are you sure you wish to continue (Y/N)
Installation Started…

Installing required packages, if this fails
make sure you have an active internet connection.

Packages to be installed: httpd php php-cli php-common php-gd php-mysql mysql mysql-server tftp-server nfs-utils vsftpd net-tools wget xinetd tar gzip make m4 gcc gcc-c++ lftp
Skipping package: httpd (Already installed)
Skipping package: php (Already installed)
Skipping package: php-cli (Already installed)
Skipping package: php-common (Already installed)
Skipping package: php-gd (Already installed)
Skipping package: php-mysql (Already installed)
Skipping package: mysql (Already installed)
Skipping package: mysql-server (Already installed)
Skipping package: tftp-server (Already installed)
Skipping package: nfs-utils (Already installed)
Skipping package: vsftpd (Already installed)
Skipping package: net-tools (Already installed)
Skipping package: wget (Already installed)
Skipping package: xinetd (Already installed)
Skipping package: tar (Already installed)
Skipping package: gzip (Already installed)
Skipping package: make (Already installed)
Skipping package: m4 (Already installed)
Skipping package: gcc (Already installed)
Skipping package: gcc-c++ (Already installed)
Skipping package: lftp (Already installed)

Confirming package installation.

Checking package: httpd…OK
Checking package: php…OK
Checking package: php-cli…OK
Checking package: php-common…OK
Checking package: php-gd…OK
Checking package: php-mysql…OK
Checking package: mysql…OK
Checking package: mysql-server…OK
Checking package: tftp-server…OK
Checking package: nfs-utils…OK
Checking package: vsftpd…OK
Checking package: net-tools…OK
Checking package: wget…OK
Checking package: xinetd…OK
Checking package: tar…OK
Checking package: gzip…OK
Checking package: make…OK
Checking package: m4…OK
Checking package: gcc…OK
Checking package: gcc-c++…OK
Checking package: lftp…OK

Configuring services.

What is the storage location for your images directory? (/images)  * Setting up and starting MySQL...OK

Backing up user reports…OK
Setting up and starting Apache Web Server…OK
Changing permissions on apache log files…OK
Downloading kernels and inits…OK
Copying back any custom hook files.
Copying back any custom report files.
You still need to install/update your database schema.
This can be done by opening a web browser and going to:

http://192.168.1.3/fog/management

Press [Enter] key when database is updated/installed.

Setting up storage…OK
Setting up and starting DHCP Server…Skipped
Setting up and starting TFTP and PXE Servers…OK
Setting up and starting VSFTP Server…OK
Setting up sudo settings…OK
Setting up FOG Snapins…OK
Setting up and building UDPCast…OK
Installing init scripts…OK
Setting up FOG Services…OK
Setting up FOG Utils…OK
Starting FOG Multicast Management Server…OK
Starting FOG Image Replicator Server…OK
Starting FOG Task Scheduler Server…OK
Starting FOG Snapin Replicator Server…OK
Configuring Fresh Clam…OK
Setting up and starting NFS Server…OK

Setup complete!

You can now login to the FOG Management Portal using
the information listed below. The login information
is only if this is the first install.

This can be done by opening a web browser and going to:

  http://192.168.1.3/fog/management

  Default User:
        Username: fog
        Password: password

[/CODE]

Sebastian Roth · Jan 30, 2015, 2:48 AM

Are all FOG services on the server running??
[CODE]ps aux | grep FOG[/CODE]

If not, restart all of them (sometimes you even need to restart them twice):
[CODE]service FOGImageReplicator restart
service FOGScheduler restart
service FOGMulticastManager restart[/CODE]

arainero · Jan 30, 2015, 3:28 AM

Looks like everything is running:

[CODE][root@fogserv ~]# ps aux | grep FOG
root 983 0.0 0.0 103252 824 pts/1 S+ 22:27 0:00 grep FOG
root 9100 0.0 0.1 251496 11216 pts/0 S 19:59 0:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FOGImageReplicator
root 9146 0.0 0.1 252008 11800 pts/0 S 19:59 0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
root 9181 0.0 0.1 251240 11028 pts/0 S 19:59 0:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
root 10669 0.0 0.1 251756 11692 pts/0 S 20:17 0:02 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
[/CODE]

Wolfbane8653 · Jan 30, 2015, 1:01 PM

[LIST]
[]What are your results when testing the 1 client troubleshooting guide?[url]http://www.fogproject.org/wiki/index.php/Troubleshooting_a_multicast#Testing_1_Client[/url]
[]What are your results when testing the 2 client troubleshooting guide? [url]http://www.fogproject.org/wiki/index.php/Troubleshooting_a_multicast#Testing_2_Clients[/url]
[/LIST]
The results here really do matter in narrowing down the issue.

Sebastian Roth · Jan 30, 2015, 4:26 PM

[quote=“arainero, post: 41506, member: 27492”]I also just noticed this populating in the bash prompt:

[CODE][root@fogserv bin]#
PHP Warning: in_array() expects parameter 2 to be array, null given in /var/www/html/fog/lib/fog/MulticastManager.class.php on line 16
PHP Warning: natsort() expects parameter 1 to be array, null given in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 152
PHP Warning: Invalid argument supplied for foreach() in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 153
PHP Warning: implode(): Argument must be an array in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 155
PHP Warning: in_array() expects parameter 2 to be array, null given in /var/www/html/fog/lib/fog/MulticastManager.class.php on line 16
PHP Warning: natsort() expects parameter 1 to be array, null given in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 152
PHP Warning: Invalid argument supplied for foreach() in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 153
PHP Warning: implode(): Argument must be an array in /var/www/html/fog/lib/fog/MulticastTask.class.php on line 155
[/CODE][/quote]

I am still very new to this project and don’t have a deep insight into the PHP code yet. But taking a look at those lines where these warnings occoured it seams like the image files are missing or can’t be read/found!? Where are your images? What do you see running this command:
[CODE]ls -al /images/*[/CODE]

arainero · Jan 30, 2015, 4:33 PM

I have done both tests before with no problems, but I will do them again.

I had a successful test 1:

[CODE][root@fogserv bin]# udp-sender --file /opt/fog/.fogsettings --log /opt/fog/log/multicast.log --ttl 1 --nopointopoint
Udp-sender 20120424
Using full duplex mode
New connection from 192.168.1.142 (#0) 00000009
Ready. Press any key to start sending data.
[/CODE]
[IMG]http://i.imgur.com/pTsIG6t.jpg[/IMG]

Test 2 worked as well. I want to point out that on the wiki it says “udb-receiver” where i imagine it’s supposed to be “udp-receiver”.

[CODE][root@fogserv bin]# udp-sender --file /opt/fog/.fogsettings --log /opt/fog/log/multicast.log --ttl 32 --nopointopoint
Udp-sender 20120424
Using full duplex mode
New connection from 192.168.1.141 (#0) 00000009
Ready. Press any key to start sending data.
New connection from 192.168.1.142 (#1) 00000009
Ready. Press any key to start sending data.
[root@fogserv bin]# re-xmits=0000000 ( 0.0%) slice=0112 - 0
[/CODE]

Here is what was added to the multicast log:

[CODE]11:21:40.186529 Using mcast address 232.168.1.3
11:21:40.186615 UDP sender for /opt/fog/.fogsettings at 192.168.1.3 on eth0
11:21:40.186631 Broadcasting control to 192.168.1.255
[01-30-15 4:21:40 pm] * No Tasks Found!
[01-30-15 4:21:50 pm] * No Tasks Found!
[01-30-15 4:22:00 pm] * No Tasks Found!
[01-30-15 4:22:10 pm] * No Tasks Found!
[01-30-15 4:22:20 pm] * No Tasks Found!
[01-30-15 4:22:30 pm] * No Tasks Found!
11:22:37.110626 Starting transfer: 00000009
11:22:37.123028 Transfer complete.
11:22:37.123072 Disconnecting #0 (192.168.1.142)
11:22:37.123107
[01-30-15 4:22:40 pm] * No Tasks Found!
[01-30-15 4:22:50 pm] * No Tasks Found!
[01-30-15 4:23:00 pm] * No Tasks Found!
[01-30-15 4:23:10 pm] * No Tasks Found!
[01-30-15 4:23:20 pm] * No Tasks Found!
[01-30-15 4:23:30 pm] * No Tasks Found!
[01-30-15 4:23:40 pm] * No Tasks Found!
[01-30-15 4:23:50 pm] * No Tasks Found!
[01-30-15 4:24:00 pm] * No Tasks Found!
[01-30-15 4:24:11 pm] * No Tasks Found!
[01-30-15 4:24:21 pm] * No Tasks Found!
[01-30-15 4:24:31 pm] * No Tasks Found!
[01-30-15 4:24:41 pm] * No Tasks Found!
[01-30-15 4:24:51 pm] * No Tasks Found!
[01-30-15 4:25:01 pm] * No Tasks Found!
[01-30-15 4:25:11 pm] * No Tasks Found!
[01-30-15 4:25:21 pm] * No Tasks Found!
[01-30-15 4:25:31 pm] * No Tasks Found!
[01-30-15 4:25:41 pm] * No Tasks Found!
[01-30-15 4:25:51 pm] * No Tasks Found!
[01-30-15 4:26:01 pm] * No Tasks Found!
[01-30-15 4:26:11 pm] * No Tasks Found!
[01-30-15 4:26:21 pm] * No Tasks Found!
[01-30-15 4:26:31 pm] * No Tasks Found!
11:26:34.969640 Using mcast address 232.168.1.3
11:26:34.969731 UDP sender for /opt/fog/.fogsettings at 192.168.1.3 on eth0
11:26:34.969747 Broadcasting control to 224.0.0.1
[01-30-15 4:26:41 pm] * No Tasks Found!
[01-30-15 4:26:51 pm] * No Tasks Found!
[01-30-15 4:27:01 pm] * No Tasks Found!
[01-30-15 4:27:11 pm] * No Tasks Found!
[01-30-15 4:27:21 pm] * No Tasks Found!
[01-30-15 4:27:31 pm] * No Tasks Found!
[01-30-15 4:27:41 pm] * No Tasks Found!
[01-30-15 4:27:51 pm] * No Tasks Found!
[01-30-15 4:28:01 pm] * No Tasks Found!
11:28:03.473599 Starting transfer: 00000009
11:28:03.475066 Transfer complete.
11:28:03.475107 Disconnecting #0 (192.168.1.141)
11:28:03.475134 Disconnecting #1 (192.168.1.142)
11:28:03.475162
[01-30-15 4:28:11 pm] * No Tasks Found![/CODE]

I would also like to add that we have a pfSense router and two Cisco Meraki switches. I do not believe they are causing problems since multicast has worked before, but I thought you should know.

arainero · Jan 30, 2015, 4:39 PM

[quote=“Uncle Frank, post: 41525, member: 28116”]I am still very new to this project and don’t have a deep insight into the PHP code yet. But taking a look at those lines where these warnings occoured it seams like the image files are missing or can’t be read/found!? Where are your images? What do you see running this command:
[CODE]ls -al /images/*[/CODE][/quote]

I get:

[CODE]/images/11515:
total 620975000
drwxrwxrwx 2 root root 4096 Jan 16 00:32 .
drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …
-rwxrwxrwx 1 root root 512 Jan 16 00:11 d1.mbr
-rwxrwxrwx 1 root root 0 Jan 16 00:11 d1.original.swapuuids
-rwxrwxrwx 1 root root 8522347 Jan 16 00:11 d1p1.img
-rwxrwxrwx 1 root root 23625716517 Jan 16 00:32 d1p2.img
-rwxrwxrwx 1 root root 512 Jan 16 00:32 d2.mbr
-rwxrwxrwx 1 root root 0 Jan 16 00:32 d2.original.swapuuids
-rwxrwxrwx 1 root root 612244074192 Jan 16 05:35 d2p1.img

/images/12815image:
total 8
drwxrwxrwx 2 root root 4096 Jan 29 18:17 .
drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …

/images/12817image:
total 624316568
drwxrwxrwx 2 root root 4096 Jan 29 18:51 .
drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …
-rwxrwxrwx 1 root root 512 Jan 28 21:49 d1.mbr
-rwxrwxrwx 1 root root 8520325 Jan 28 21:49 d1p1.img
-rwxrwxrwx 1 root root 24696905617 Jan 28 22:09 d1p2.img
-rwxrwxrwx 1 root root 512 Jan 28 22:09 d2.mbr
-rwxrwxrwx 1 root root 614594652526 Jan 29 03:17 d2p1.img

/images/dev:
total 16
drwxrwxrwx. 4 root root 4096 Jan 29 18:15 .
drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …
drwxrwxrwx 2 root root 4096 Jan 16 14:05 fcaa14310adb
drwxrwxrwx 2 root root 4096 Jan 29 18:15 fcaa14310c42
-rwxrwxrwx. 1 root root 0 Jan 15 22:15 .mntcheck

/images/postdownloadscripts:
total 12
drwxrwxrwx. 2 root root 4096 Jan 15 22:15 .
drwxrwxrwx. 7 root root 4096 Jan 29 18:51 …
-rwxrwxrwx. 1 root root 233 Jan 15 22:15 fog.postdownload
[/CODE]

In the process of me removing FOG and reinstalling it in attempts to get it to work I followed the following from the wiki: [url]http://www.fogproject.org/wiki/index.php/Uninstall_FOG[/url]. EXCEPT deleting the images folder. I noticed that when I reinstalled FOG that my images didn’t show up until I created a “new” image and then I moved the old files into the newly created folder. I believe a database entry had to be made for them to show up.

I’d like to mention that I am able to do a Unicast just fine, so I don’t believe there is a problem reading the image. Maybe multicast reads it differently though?

Tom Elliott · Jan 30, 2015, 6:24 PM

SVN 2951 should fix this issue for you.

Wolfbane8653 · Jan 31, 2015, 4:18 AM

[quote=“arainero, post: 41526, member: 27492”]
Test 2 worked as well. I want to point out that on the wiki it says “udb-receiver” where i imagine it’s supposed to be “udp-receiver”.
[/quote]

wiki has been fixed thanks for reporting!

arainero · Jan 31, 2015, 7:21 PM

The changes Tom made corrected the issue. Thanks again!