Multicast not starting

[gerald.stachl]

@Tom-Elliott

Yes, there are a lot of error messages. 10.2.2.83 is the IP of the client i try to multicast. Seems that there are problems with DB connections? - (Correction: i saw, that these errors are from yesterday: maybe when i truncated the mysql multicast and tasks tables?)
-> fogcontroller.class.php on line 197 ??

[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::select_db(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 38
[Sat May 14 16:24:12 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::query(): Couldn’t fetch mysqli in /var/www/fog/lib/db/mysql.class.php on line 50
[Sat May 14 16:24:15 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::mysqli(): MySQL server has gone away in /var/www/fog/lib/db/mysql.class.php on line 25
[Sat May 14 16:24:15 2016] [error] [client 10.2.2.83] PHP Warning: mysqli::mysqli(): MySQL server has gone away in /var/www/fog/lib/db/mysql.class.php on line 28
[Sat May 14 16:24:16 2016] [notice] caught SIGTERM, shutting down
[Sat May 14 16:25:13 2016] [notice] Apache/2.2.22 (Debian) PHP/5.6.21-1~dotdeb+7.1 mod_ssl/2.2.22 OpenSSL/1.0.1e configured – resuming normal operations
[Sat May 14 16:25:25 2016] [error] [client 10.2.0.26] PHP Warning: Cannot modify header information - headers already sent by (output started at /var/www/fog/lib/fog/schema.class.php:46) in /var/www/fog/lib/fog/reportmaker.class.php on line 98
[Sat May 14 16:29:05 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sat May 14 16:29:05 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sat May 14 16:29:05 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sat May 14 16:29:49 2016] [error] [client 10.2.2.83] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197
[Sat May 14 16:29:59 2016] [error] [client 10.2.2.83] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197
[Sun May 15 06:39:39 2016] [notice] caught SIGTERM, shutting down
[Sun May 15 06:40:15 2016] [notice] Apache/2.2.22 (Debian) PHP/5.6.21-1~dotdeb+7.1 mod_ssl/2.2.22 OpenSSL/1.0.1e configured – resuming normal operations
[Sun May 15 06:45:08 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sun May 15 06:45:08 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8
[Sun May 15 06:45:08 2016] [error] [client 10.1.1.10] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197, referer: http://10.2.0.26/fog/management/index.php?node=group&sub=deploy&id=6&type=8

[gerald.stachl]

@Tom-Elliott
i just found something in syslog:

• not sure if this is correct. The errors above come from samba, because SAMBA4 is also installed on this server.

[Tom Elliott]

@gerald.stachl rpc is nfs. tftp is well, tftp. Can you update to the latest, again, please.

[gerald.stachl]

@Tom-Elliott
Did the Update (7701)

At the moment there is an error in management interface

[Tom Elliott]

@gerald.stachl Update again, but also do a hard refresh on the page.

[gerald.stachl]

@Tom-Elliott
Did both, the error on the task management page is gone, but still the same error when trying to start multicast.

[Sebastian Roth]

@gerald-stachl said:

same error when trying to start multicast

Which error? The ones you posted from the apache log don’t seem to be related to multicast!

[gerald.stachl]

@Sebastian-Roth
I mean, that multicast is not starting. In apache error log there are lines like “[client 10.2.2.83] PHP Warning: array_map(): An error occurred while invoking the map callback in /var/www/fog/lib/fog/fogcontroller.class.php on line 197”
There is another thread with similar error in logs:
https://forums.fogproject.org/topic/7412/problems-with-multicast-task/23

[Sebastian Roth]

@gerald-stachl Ok, thanks for clarifying! I am calling @Tom-Elliott as I refuse to try to debug this kind of code (nested array_map calls). Sorry Tom, but this will be all yours…

[Tom Elliott]

@Sebastian-Roth Error throwing is due to array_map not being able to handle “throw new” from within. I’ve adjusted all functional programming within the fogcontroller.class.php file to array_walk in hopes to fix the issue. I don’t know if it will fix the “mutlicast not starting” issue.

[gerald.stachl]

@Tom-Elliott
The array_map() errors are gone now, but multicast is still not starting:

I can’t find any errors in apache logs now. I also wonder, why there is no multicast.log file? (i had a file, but this file was created during test with udp-sender on the console)
I cleared the folder /opt/fog/log and ran the installer again. After that the Service Master Logfile is created new, but all the other logfiles don’t exist:

in the Service Master Log i can see:
[05-19-16 4:40:11 am] FOGPingHosts child process (27579) exited with code 255.
[05-19-16 4:40:09 am] FOGTaskScheduler child process (27578) exited with code 255.
[05-19-16 4:40:07 am] FOGImageReplicator child process (27577) exited with code 255.
[05-19-16 4:40:05 am] FOGImageReplicator child process (27576) exited with code 255.
[05-19-16 4:40:03 am] FOGMulticastManager child process (27575) exited with code 255.

(is that OK?) - The timezone in the logfiles seems to be UTC, even if i correct the timezone settings in php.ini. The system time (date) also shows the correct time.

Any other debug options i could use?

[Sebastian Roth]

@gerald.stachl said in Multicast not starting:

exited with code 255.

Definitely does not sound too good. As well you should have other logs. Is this Ubuntu? Did you restart the whole system? There is a known issue with the services starting too early before the mysql DB is properly up. Please read Tom’s great post about this here: https://forums.fogproject.org/topic/3081/fog-1-1-0-multicast-sits-at-starting-to-restore-image-to-device-dev-sda1/78

[gerald.stachl]

@Sebastian-Roth
My system is Ubuntu 14.04 LTS. The thread is for systemd startup, while my system uses sysvinit.
i have a strange behaviour now: After a system restart none of the FOG Service is running. When i try to start them with “service FOGMulticastManager start” i get an error. “service FOGMulticastManager restart” works!! After that the service seems to be running.

root@imaging:~# service FOGImageReplicator restart
[…] Restarting FOG Computer Imaging Solution: FOGImageReplicatorstart-stop-daemon: warning: failed to kill 2283: No such process
. ok

And there are no logfiles! (except Service Master Log)

[Sebastian Roth]

@gerald-stachl This is known behaviour! Do the services run after restarting by hand? ps ax | grep FOG? Still no logs???

[gerald.stachl]

@Sebastian-Roth
Yes the service are up, but there are still no log files.

root@imaging:~# ps ax |grep FOG
4905 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
5014 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FOGImageReplicator
6027 ? Z 0:00 [FOGMulticastMan] <defunct>
6029 ? Z 0:00 [FOGImageReplica] <defunct>
6127 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
6143 ? Z 0:00 [FOGPingHosts] <defunct>
6160 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
6177 ? Z 0:00 [FOGTaskSchedule] <defunct>
6199 ? S 0:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
6217 ? Z 0:00 [FOGSnapinReplic] <defunct>
6219 pts/0 R+ 0:00 grep FOG
root@imaging:~#

[Tom Elliott]

@gerald.stachl can you stop the services, then start them after these have been stopped for a few seconds?

Restart works kind of but is often too fast to actually obtain the info as needed as essentially two services, for a small period of time, overlap.

[gerald.stachl]

@Tom-Elliott
Back again after upgrade:
i installed php7 and ran the installer with “php_ver=‘7.0’ php_verAdds=‘-7.0’ ./installfog.sh -y”
On my system there are still no logfiles, and multicast is not starting.
Trying to start FOGMulticastManager from commandline results in same error message:

As far i can see, there are still some php5 packages: (by fog installer?)

root@imaging:/opt/fog/log# dpkg -l | grep php
ii libapache2-mod-php5 5.6.20+dfsg-0+deb8u1 amd64 server-side, HTML-embedded scripting language (Apache 2 module)
ii libapache2-mod-php7.0 7.0.6-1~dotdeb+8.1 amd64 server-side, HTML-embedded scripting language (Apache 2 module)
ii libphp7.0-embed 7.0.6-1~dotdeb+8.1 amd64 HTML-embedded scripting language (Embedded SAPI library)
ii php-common 21-1~dotdeb+8.1 all Common files for PHP packages
ii php-gettext 1.0.11-1 all read gettext MO files directly, without requiring anything other than PHP
ii php-readline 21-1~dotdeb+8.1 all readline module for PHP [default]
ii php5 5.6.20+dfsg-0+deb8u1 all server-side, HTML-embedded scripting language (metapackage)
ii php5-cli 5.6.20+dfsg-0+deb8u1 amd64 command-line interpreter for the php5 scripting language
ii php5-common 5.6.20+dfsg-0+deb8u1 amd64 Common files for packages built from the php5 source
ii php5-curl 5.6.20+dfsg-0+deb8u1 amd64 CURL module for php5
ii php5-fpm 5.6.20+dfsg-0+deb8u1 amd64 server-side, HTML-embedded scripting language (FPM-CGI binary)
ii php5-gd 5.6.20+dfsg-0+deb8u1 amd64 GD module for php5
ii php5-json 1.3.6-1 amd64 JSON module for php5
ii php5-mcrypt 5.6.20+dfsg-0+deb8u1 amd64 MCrypt module for php5
ii php5-mysqlnd 5.6.20+dfsg-0+deb8u1 amd64 MySQL module for php5 (Native Driver)
ii php5-readline 5.6.20+dfsg-0+deb8u1 amd64 Readline module for php5
ii php7.0 7.0.6-1~dotdeb+8.1 all server-side, HTML-embedded scripting language (metapackage)
ii php7.0-apcu 5.1.4-1~dotdeb+8.1 amd64 APC User Cache for PHP
ii php7.0-apcu-bc 1.0.3-1~dotdeb+8.1 amd64 APCu Backwards Compatibility Module
ii php7.0-bz2 7.0.6-1~dotdeb+8.1 amd64 bzip2 module for PHP
ii php7.0-cgi 7.0.6-1~dotdeb+8.1 amd64 server-side, HTML-embedded scripting language (CGI binary)
ii php7.0-cli 7.0.6-1~dotdeb+8.1 amd64 command-line interpreter for the PHP scripting language
ii php7.0-common 7.0.6-1~dotdeb+8.1 all Common files for packages built from the PHP source
ii php7.0-curl 7.0.6-1~dotdeb+8.1 amd64 CURL module for PHP
ii php7.0-dbg 7.0.6-1~dotdeb+8.1 amd64 Debug symbols for PHP7.0
ii php7.0-dev 7.0.6-1~dotdeb+8.1 amd64 Files for PHP7.0 module development
ii php7.0-enchant 7.0.6-1~dotdeb+8.1 amd64 Enchant module for PHP
ii php7.0-fpm 7.0.6-1~dotdeb+8.1 amd64 server-side, HTML-embedded scripting language (FPM-CGI binary)
ii php7.0-gd 7.0.6-1~dotdeb+8.1 amd64 GD module for PHP
ii php7.0-geoip 1.1.0-4~dotdeb+8.2 amd64 GeoIP module for PHP
ii php7.0-gmp 7.0.6-1~dotdeb+8.1 amd64 GMP module for PHP
ii php7.0-igbinary 1.2.1-1~2b7c703-1~dotdeb+8.1 amd64 igbinary serializer for PHP
ii php7.0-imagick 3.4.2-1~dotdeb+8.1 amd64 Provides a wrapper to the ImageMagick library
ii php7.0-imap 7.0.6-1~dotdeb+8.1 amd64 IMAP module for PHP
ii php7.0-interbase 7.0.6-1~dotdeb+8.1 amd64 Interbase module for PHP
ii php7.0-intl 7.0.6-1~dotdeb+8.1 amd64 Internationalisation module for PHP
ii php7.0-json 7.0.6-1~dotdeb+8.1 amd64 JSON module for PHP
ii php7.0-ldap 7.0.6-1~dotdeb+8.1 amd64 LDAP module for PHP
ii php7.0-mcrypt 7.0.6-1~dotdeb+8.1 amd64 libmcrypt module for PHP
ii php7.0-memcached 2.2.0-2-52c644b-1~dotdeb+8.1 amd64 memcached extension module for PHP, uses libmemcached
ii php7.0-mongodb 1.1.6-1~dotdeb+8.1 amd64 MongoDB driver for PHP
ii php7.0-msgpack 2.0.1-1~dotdeb+8.1 amd64 MessagePack serializer for PHP
ii php7.0-mysql 7.0.6-1~dotdeb+8.1 amd64 MySQL module for PHP
ii php7.0-odbc 7.0.6-1~dotdeb+8.1 amd64 ODBC module for PHP
ii php7.0-opcache 7.0.6-1~dotdeb+8.1 amd64 Zend OpCache module for PHP
ii php7.0-pgsql 7.0.6-1~dotdeb+8.1 amd64 PostgreSQL module for PHP
ii php7.0-phpdbg 7.0.6-1~dotdeb+8.1 amd64 server-side, HTML-embedded scripting language (PHPDBG binary)
ii php7.0-pspell 7.0.6-1~dotdeb+8.1 amd64 pspell module for PHP
ii php7.0-readline 7.0.6-1~dotdeb+8.1 amd64 readline module for PHP
ii php7.0-recode 7.0.6-1~dotdeb+8.1 amd64 recode module for PHP
ii php7.0-redis 2.2.7-1~7b36957-1~dotdeb+8.1 amd64 PHP extension for interfacing with Redis
ii php7.0-snmp 7.0.6-1~dotdeb+8.1 amd64 SNMP module for PHP
ii php7.0-sqlite3 7.0.6-1~dotdeb+8.1 amd64 SQLite3 module for PHP
ii php7.0-sybase 7.0.6-1~dotdeb+8.1 amd64 Sybase module for PHP
ii php7.0-tidy 7.0.6-1~dotdeb+8.1 amd64 tidy module for PHP
ii php7.0-xdebug 1:2.4.0-1~dotdeb+8.1 amd64 Xdebug Module for PHP
ii php7.0-xmlrpc 7.0.6-1~dotdeb+8.1 amd64 XMLRPC-EPI module for PHP
ii php7.0-xsl 7.0.6-1~dotdeb+8.1 amd64 XSL module for PHP

[gerald.stachl]

@gerald.stachl
Seems. that the problem is solved:

I found another thread: https://forums.fogproject.org/topic/6129/multicasting-stuck-on-starting-to-restore-image/36 where Tom gave the advice to remove the directory /opt/fog and start the installer again.
I did that, and after that i saw, that all the logfiles are created successfully. Then i cleared the old tasks

mysql -u root
use fog;
truncate multicastSessions;
truncate multicastSessionsAssoc;
truncate tasks;

after that i rebooted the server. Now newly created Multicasttasks work!

Thanks for your help!

My Multicast Tasks are starting now, but (i tried it 3 times with different images (all of them working with unicast)) they are not finishing => shall i open a new thread and mark this one as solved?

EDIT: Did other tests with multicast. At first i tried a small image (Zorin Linux - 8GB) it worked. After that i tried one of our Windows 10 Images again (~70GB) and it worked also.
So i mark this case with SOLVED!

[habu]

@gerald.stachl

I solved this question, i hope it may help someone.

• FOG 1.3-RC1 on CentOS 7.0.
• Unicast deploy working.
• Even multicast was working.

After a system reboot, MulticastManager didn’t start again:

Sep 4 06:01:28 clonacion FOGMulticastManager: PHP Fatal error: Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 13

It didn’t matter if I added a multicast task: udp-sender process didn’t start because service didn’t start before (you’ll see only one FOGMulticastManager in process list --ps–, it’s the parent which forks periodically one child process (but this one will die at GetBanned method invocation).

If I commented GetBanner() invocation out at /opt/fog/service/FOGMulticastManager/FOGMulticastManager, then it failed at next method call to object MulticastManager.

I found TWO “MulticastManager” class, one of them extends FOGBase (at lib/fog directory) and the other one extends FOGService (at lib/service directory). Only the last one has defined methods like GetBanner(), etc. By any reason, it uses in some cases wrongly the first one (even when it used the second one after installation because it worked before).

So, I renamed MulticastManager class that extends FOGService to MulticastManagerService, and I changed the code to use that. After that, service FOGMulticastService starts with no problems, it works. I did the same thing with rest of services: FOGScheduler,…

I resume (for service MulticastManager: repeat it with other services in /opt/fog/services directory):

1. Edit /opt/fog/service/FOGMulticastManager/FogMulticastManager file: modify line 8 from $ServiceClass = FOGCore::getClass(‘MulticastManage’); to $ServiceClass = FOGCore::getClass(‘MulticastManagerService’);

2. Rename file /var/www/html/fog/lib/service/multicastmanager.class.php to /var/www/html/fog/lib/service/multicastmanagerservice.class.php (/var/www/html/fog is the path to fog web service)

3. Edit /var/www/html/fog/lib/service/multicastmanagerservice.class.php and modify the name of this class (line 2) from “class MulticastManager extends FOGService” to “class MulticastManagerService extends FOGService”