Multicast not starting

[gerald.stachl]

@Sebastian-Roth
root@imaging:~#
ps ax|grep FOG
49456 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FOGImageReplicator
49467 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
49478 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
49489 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
49760 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
52203 ? Z 0:00 [FOGMulticastMan] <defunct>
52267 pts/0 S+ 0:00 grep FOG

i killed ps 52203 and then i rebooted the server. After that i got:

2271 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FOGImageReplicator
2324 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
2382 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
2411 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
2441 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
4428 ? Z 0:00 [FOGMulticastMan] <defunct>
4429 ? Z 0:00 [FOGImageReplica] <defunct>
4430 ? Z 0:00 [FOGPingHosts] <defunct>
4431 ? Z 0:00 [FOGTaskSchedule] <defunct>
4432 ? Z 0:00 [FOGSnapinReplic] <defunct>
4444 pts/0 S+ 0:00 grep FOG

sometimes the zombie processes disappear:
2271 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FOGImageReplicator
2324 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
2382 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
2411 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
2441 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
4868 pts/0 S+ 0:00 grep FOG

then they appear again.

On this server, unicast imaging works fine. I had some problems with the FOG Client - i had to change back to the legacy client, because the domain join didn’t work with the new client (but it worked some weeks ago - but this might be a different problem)

[Tom Elliott]

@gerald.stachl is the OS this is installed using systemd or sysvinit? If systemctl can you get output of systemctl -l status FOGMulticastManager? If sysvinit can you get any information from the Apache error logs?

[gerald.stachl]

@Tom-Elliott

Yes, there are a lot of error messages. 10.2.2.83 is the IP of the client i try to multicast. Seems that there are problems with DB connections? - (Correction: i saw, that these errors are from yesterday: maybe when i truncated the mysql multicast and tasks tables?)
-> fogcontroller.class.php on line 197 ??

[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:15 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::mysqli(): MySQL server has gone away in /var/www/fog/lib/db/mysql.class.php on line 25
[Sat May 14 16:24:15 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::mysqli(): MySQL server has gone away in /var/www/fog/lib/db/mysql.class.php on line 28
[Sat May 14 16:24:16 2016] [notice] caught SIGTERM, shutting down
[Sat May 14 16:25:13 2016] [notice] Apache/2.2.22 (Debian) PHP/5.6.21-1~dotdeb+7.1 mod_ssl/2.2.22 OpenSSL/1.0.1e configured – resuming normal operations
[Sat May 14 16:25:25 2016] [error] [client 10.2.0.26] PHP Warning: Cannot modify header information - headers already sent by (output started at /var/www/fog/lib/fog/schema.class.php:46) in /var/www/fog/lib/fog/reportmaker.class.php on line 98
[Sat May 14 16:29:05 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sat May 14 16:29:05 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sat May 14 16:29:05 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sat May 14 16:29:49 2016] [error] [client 10.2.2.83] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197
[Sat May 14 16:29:59 2016] [error] [client 10.2.2.83] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197
[Sun May 15 06:39:39 2016] [notice] caught SIGTERM, shutting down
[Sun May 15 06:40:15 2016] [notice] Apache/2.2.22 (Debian) PHP/5.6.21-1~dotdeb+7.1 mod_ssl/2.2.22 OpenSSL/1.0.1e configured – resuming normal operations
[Sun May 15 06:45:08 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sun May 15 06:45:08 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sun May 15 06:45:08 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8

[gerald.stachl]

@Tom-Elliott
i just found something in syslog:

• not sure if this is correct. The errors above come from samba, because SAMBA4 is also installed on this server.

[Tom Elliott]

@gerald.stachl rpc is nfs. tftp is well, tftp. Can you update to the latest, again, please.

[gerald.stachl]

@Tom-Elliott
Did the Update (7701)

At the moment there is an error in management interface

[Tom Elliott]

@gerald.stachl Update again, but also do a hard refresh on the page.

[gerald.stachl]

@Tom-Elliott
Did both, the error on the task management page is gone, but still the same error when trying to start multicast.

[Sebastian Roth]

@gerald-stachl said:

same error when trying to start multicast

Which error? The ones you posted from the apache log don’t seem to be related to multicast!

[gerald.stachl]

@Sebastian-Roth
I mean, that multicast is not starting. In apache error log there are lines like “[client 10.2.2.83] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197”
There is another thread with similar error in logs:
https://forums.fogproject.org/topic/7412/problems-with-multicast-task/23

[Sebastian Roth]

@gerald-stachl Ok, thanks for clarifying! I am calling @Tom-Elliott as I refuse to try to debug this kind of code (nested array_map calls). Sorry Tom, but this will be all yours…

[Tom Elliott]

@Sebastian-Roth Error throwing is due to array_map not being able to handle “throw new” from within. I’ve adjusted all functional programming within the fogcontroller.class.php file to array_walk in hopes to fix the issue. I don’t know if it will fix the “mutlicast not starting” issue.

[gerald.stachl]

@Tom-Elliott
The array_map() errors are gone now, but multicast is still not starting:

I can’t find any errors in apache logs now. I also wonder, why there is no multicast.log file? (i had a file, but this file was created during test with udp-sender on the console)
I cleared the folder /opt/fog/log and ran the installer again. After that the Service Master Logfile is created new, but all the other logfiles don’t exist:

in the Service Master Log i can see:
[05-19-16 4:40:11 am] FOGPingHosts child process (27579) exited with code 255.
[05-19-16 4:40:09 am] FOGTaskScheduler child process (27578) exited with code 255.
[05-19-16 4:40:07 am] FOGImageReplicator child process (27577) exited with code 255.
[05-19-16 4:40:05 am] FOGImageReplicator child process (27576) exited with code 255.
[05-19-16 4:40:03 am] FOGMulticastManager child process (27575) exited with code 255.

(is that OK?) - The timezone in the logfiles seems to be UTC, even if i correct the timezone settings in php.ini. The system time (date) also shows the correct time.

Any other debug options i could use?

[Sebastian Roth]

@gerald.stachl said in Multicast not starting:

exited with code 255.

Definitely does not sound too good. As well you should have other logs. Is this Ubuntu? Did you restart the whole system? There is a known issue with the services starting too early before the mysql DB is properly up. Please read Tom’s great post about this here: https://forums.fogproject.org/topic/3081/fog-1-1-0-multicast-sits-at-starting-to-restore-image-to-device-dev-sda1/78

[gerald.stachl]

@Sebastian-Roth
My system is Ubuntu 14.04 LTS. The thread is for systemd startup, while my system uses sysvinit.
i have a strange behaviour now: After a system restart none of the FOG Service is running. When i try to start them with “service FOGMulticastManager start” i get an error. “service FOGMulticastManager restart” works!! After that the service seems to be running.

root@imaging:~# service FOGImageReplicator restart
[…] Restarting FOG Computer Imaging Solution: FOGImageReplicatorstart-stop-daemon: warning: failed to kill 2283: No such process
. ok

And there are no logfiles! (except Service Master Log)

[Sebastian Roth]

@gerald-stachl This is known behaviour! Do the services run after restarting by hand? ps ax | grep FOG? Still no logs???

[gerald.stachl]

@Sebastian-Roth
Yes the service are up, but there are still no log files.

root@imaging:~# ps ax |grep FOG
4905 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
5014 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FOGImageReplicator
6027 ? Z 0:00 [FOGMulticastMan] <defunct>
6029 ? Z 0:00 [FOGImageReplica] <defunct>
6127 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
6143 ? Z 0:00 [FOGPingHosts] <defunct>
6160 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
6177 ? Z 0:00 [FOGTaskSchedule] <defunct>
6199 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
6217 ? Z 0:00 [FOGSnapinReplic] <defunct>
6219 pts/0 R+ 0:00 grep FOG
root@imaging:~#

[Tom Elliott]

@gerald.stachl can you stop the services, then start them after these have been stopped for a few seconds?

Restart works kind of but is often too fast to actually obtain the info as needed as essentially two services, for a small period of time, overlap.

[gerald.stachl]

@Tom-Elliott
Back again after upgrade:
i installed php7 and ran the installer with “php_ver=‘7.0’ php_verAdds=‘-7.0’ ./installfog.sh -y”
On my system there are still no logfiles, and multicast is not starting.
Trying to start FOGMulticastManager from commandline results in same error message:

As far i can see, there are still some php5 packages: (by fog installer?)

root@imaging:/opt/fog/log# dpkg -l | grep php
ii libapache2-mod-php5 5.6.20+dfsg-0+deb8u1 amd64 server-side, HTML-embedded scripting language (Apache 2 module)
ii libapache2-mod-php7.0 7.0.6-1~dotdeb+8.1 amd64 server-side, HTML-embedded scripting language (Apache 2 module)
ii libphp7.0-embed 7.0.6-1~dotdeb+8.1 amd64 HTML-embedded scripting language (Embedded SAPI library)
ii php-common 21-1~dotdeb+8.1 all Common files for PHP packages
ii php-gettext 1.0.11-1 all read gettext MO files directly, without requiring anything other than PHP
ii php-readline 21-1~dotdeb+8.1 all readline module for PHP [default]
ii php5 5.6.20+dfsg-0+deb8u1 all server-side, HTML-embedded scripting language (metapackage)
ii php5-cli 5.6.20+dfsg-0+deb8u1 amd64 command-line interpreter for the php5 scripting language
ii php5-common 5.6.20+dfsg-0+deb8u1 amd64 Common files for packages built from the php5 source
ii php5-curl 5.6.20+dfsg-0+deb8u1 amd64 CURL module for php5
ii php5-fpm 5.6.20+dfsg-0+deb8u1 amd64 server-side, HTML-embedded scripting language (FPM-CGI binary)
ii php5-gd 5.6.20+dfsg-0+deb8u1 amd64 GD module for php5
ii php5-json 1.3.6-1 amd64 JSON module for php5
ii php5-mcrypt 5.6.20+dfsg-0+deb8u1 amd64 MCrypt module for php5
ii php5-mysqlnd 5.6.20+dfsg-0+deb8u1 amd64 MySQL module for php5 (Native Driver)
ii php5-readline 5.6.20+dfsg-0+deb8u1 amd64 Readline module for php5
ii php7.0 7.0.6-1~dotdeb+8.1 all server-side, HTML-embedded scripting language (metapackage)
ii php7.0-apcu 5.1.4-1~dotdeb+8.1 amd64 APC User Cache for PHP
ii php7.0-apcu-bc 1.0.3-1~dotdeb+8.1 amd64 APCu Backwards Compatibility Module
ii php7.0-bz2 7.0.6-1~dotdeb+8.1 amd64 bzip2 module for PHP
ii php7.0-cgi 7.0.6-1~dotdeb+8.1 amd64 server-side, HTML-embedded scripting language (CGI binary)
ii php7.0-cli 7.0.6-1~dotdeb+8.1 amd64 command-line interpreter for the PHP scripting language
ii php7.0-common 7.0.6-1~dotdeb+8.1 all Common files for packages built from the PHP source
ii php7.0-curl 7.0.6-1~dotdeb+8.1 amd64 CURL module for PHP
ii php7.0-dbg 7.0.6-1~dotdeb+8.1 amd64 Debug symbols for PHP7.0
ii php7.0-dev 7.0.6-1~dotdeb+8.1 amd64 Files for PHP7.0 module development
ii php7.0-enchant 7.0.6-1~dotdeb+8.1 amd64 Enchant module for PHP
ii php7.0-fpm 7.0.6-1~dotdeb+8.1 amd64 server-side, HTML-embedded scripting language (FPM-CGI binary)
ii php7.0-gd 7.0.6-1~dotdeb+8.1 amd64 GD module for PHP
ii php7.0-geoip 1.1.0-4~dotdeb+8.2 amd64 GeoIP module for PHP
ii php7.0-gmp 7.0.6-1~dotdeb+8.1 amd64 GMP module for PHP
ii php7.0-igbinary 1.2.1-1~2b7c703-1~dotdeb+8.1 amd64 igbinary serializer for PHP
ii php7.0-imagick 3.4.2-1~dotdeb+8.1 amd64 Provides a wrapper to the ImageMagick library
ii php7.0-imap 7.0.6-1~dotdeb+8.1 amd64 IMAP module for PHP
ii php7.0-interbase 7.0.6-1~dotdeb+8.1 amd64 Interbase module for PHP
ii php7.0-intl 7.0.6-1~dotdeb+8.1 amd64 Internationalisation module for PHP
ii php7.0-json 7.0.6-1~dotdeb+8.1 amd64 JSON module for PHP
ii php7.0-ldap 7.0.6-1~dotdeb+8.1 amd64 LDAP module for PHP
ii php7.0-mcrypt 7.0.6-1~dotdeb+8.1 amd64 libmcrypt module for PHP
ii php7.0-memcached 2.2.0-2-52c644b-1~dotdeb+8.1 amd64 memcached extension module for PHP, uses libmemcached
ii php7.0-mongodb 1.1.6-1~dotdeb+8.1 amd64 MongoDB driver for PHP
ii php7.0-msgpack 2.0.1-1~dotdeb+8.1 amd64 MessagePack serializer for PHP
ii php7.0-mysql 7.0.6-1~dotdeb+8.1 amd64 MySQL module for PHP
ii php7.0-odbc 7.0.6-1~dotdeb+8.1 amd64 ODBC module for PHP
ii php7.0-opcache 7.0.6-1~dotdeb+8.1 amd64 Zend OpCache module for PHP
ii php7.0-pgsql 7.0.6-1~dotdeb+8.1 amd64 PostgreSQL module for PHP
ii php7.0-phpdbg 7.0.6-1~dotdeb+8.1 amd64 server-side, HTML-embedded scripting language (PHPDBG binary)
ii php7.0-pspell 7.0.6-1~dotdeb+8.1 amd64 pspell module for PHP
ii php7.0-readline 7.0.6-1~dotdeb+8.1 amd64 readline module for PHP
ii php7.0-recode 7.0.6-1~dotdeb+8.1 amd64 recode module for PHP
ii php7.0-redis 2.2.7-1~7b36957-1~dotdeb+8.1 amd64 PHP extension for interfacing with Redis
ii php7.0-snmp 7.0.6-1~dotdeb+8.1 amd64 SNMP module for PHP
ii php7.0-sqlite3 7.0.6-1~dotdeb+8.1 amd64 SQLite3 module for PHP
ii php7.0-sybase 7.0.6-1~dotdeb+8.1 amd64 Sybase module for PHP
ii php7.0-tidy 7.0.6-1~dotdeb+8.1 amd64 tidy module for PHP
ii php7.0-xdebug 1:2.4.0-1~dotdeb+8.1 amd64 Xdebug Module for PHP
ii php7.0-xmlrpc 7.0.6-1~dotdeb+8.1 amd64 XMLRPC-EPI module for PHP
ii php7.0-xsl 7.0.6-1~dotdeb+8.1 amd64 XSL module for PHP

[gerald.stachl]

@gerald.stachl
Seems. that the problem is solved:

I found another thread: https://forums.fogproject.org/topic/6129/multicasting-stuck-on-starting-to-restore-image/36 where Tom gave the advice to remove the directory /opt/fog and start the installer again.
I did that, and after that i saw, that all the logfiles are created successfully. Then i cleared the old tasks

mysql -u root
use fog;
truncate multicastSessions;
truncate multicastSessionsAssoc;
truncate tasks;

after that i rebooted the server. Now newly created Multicasttasks work!

Thanks for your help!

My Multicast Tasks are starting now, but (i tried it 3 times with different images (all of them working with unicast)) they are not finishing => shall i open a new thread and mark this one as solved?

EDIT: Did other tests with multicast. At first i tried a small image (Zorin Linux - 8GB) it worked. After that i tried one of our Windows 10 Images again (~70GB) and it worked also.
So i mark this case with SOLVED!