Particlone Fails to Start

Tom Elliott · Jul 21, 2016, 5:02 PM

Do you have anything in error logs?

Particularly, can you try:

service FOGMulticastManager stop; sleep 5; service FOGMulticastManager start and then get us the apache error logs and watch the main log itself?

Joe Gill · Jul 21, 2016, 5:07 PM

@Tom-Elliott
When I run:
service FOGMulticastManager stop; sleep 5; service FOGMulticastManager start

And Run:
tail -100f /var/log/fog/multicast.log

The log file does not change.

In particular which of the apache logs did you want to see?

Thanks!

Tom Elliott · Jul 21, 2016, 5:08 PM

@Joe-Gill The error log.

How are you getting root?

Joe Gill · Jul 21, 2016, 5:10 PM

@Tom-Elliott

You mean sudo -i?

Where exactly is the Apache log file found?

Tom Elliott · Jul 21, 2016, 5:11 PM

@Joe-Gill File is typically located in /var/log/apache2/error.log

Yes, I mean sudo -i.

It just seems odd that the log file is not continuing on. Starting service loop should be doing something.

Joe Gill · Jul 21, 2016, 5:12 PM

@Tom-Elliott

[Mon Jul 18 07:39:37.212110 2016] [mpm_prefork:notice] [pid 1437] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.6.23-1+deprecated+dontuse+deb.sury.org~trusty+1 OpenSSL/1.0.1f c$
[Mon Jul 18 07:39:37.212138 2016] [core:notice] [pid 1437] AH00094: Command line: '/usr/sbin/apache2'
[Mon Jul 18 13:56:08.991155 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit K in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991183 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit - in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991190 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit Y in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991196 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit H in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991203 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit J in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991209 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit - in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991215 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit T in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991221 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit H in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991227 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit X in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991233 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit - in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991239 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit W in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991245 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit H in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991251 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit H in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991257 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit - in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991271 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit W in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991277 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit T in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 13:56:08.991283 2016] [:error] [pid 6324] [client 172.16.19.142:49775] PHP Warning:  pack(): Type H: illegal hex digit T in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.523933 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit K in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.523961 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit - in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.523969 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit Y in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.523976 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit H in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.523982 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit J in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.523988 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit - in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.523994 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit T in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524001 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit H in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524007 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit X in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524013 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit - in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524019 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit W in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524026 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit H in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524038 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit H in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524045 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit - in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524051 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit W in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524056 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit T in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:24:56.524062 2016] [:error] [pid 5148] [client 172.16.19.149:49681] PHP Warning:  pack(): Type H: illegal hex digit T in /var/www/html/fog/lib/fog/fogba$
[Mon Jul 18 14:48:10.311780 2016] [mpm_prefork:notice] [pid 1437] AH00169: caught SIGTERM, shutting down
[Mon Jul 18 14:48:11.358054 2016] [mpm_prefork:notice] [pid 9188] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.6.23-1+deprecated+dontuse+deb.sury.org~trusty+1 OpenSSL/1.0.1f c$
[Mon Jul 18 14:48:11.358098 2016] [core:notice] [pid 9188] AH00094: Command line: '/usr/sbin/apache2'

Hanz · Jul 21, 2016, 5:13 PM

@Tom-Elliott

[Thu Jul 21 12:46:53.589163 2016] [:error] [pid 1624] [client 10.72.129.4:59852] PHP Warning: Packets out of order. Expected 1 received 0. Packet size=30 in /var/www/html/fog/lib/db/pdodb.class.php on line 32, referer: http://10.72.3.50/fog/management/index.php?node=home

I think this is what is happening
Could be wrong, but def an error
It will start from Server at office, but not at node, which is the log I was showing was multicast from node

Joe Gill · Jul 21, 2016, 6:24 PM

@Tom-Elliott

Any thoughts on this?

Joe Gill · Jul 21, 2016, 7:07 PM

@Hanz

Same on this end. I was showing log on Node.

Tom Elliott · Jul 21, 2016, 7:12 PM

Are you using location plugin?

Joe Gill · Jul 21, 2016, 7:21 PM

@Tom-Elliott

Not that I’m aware of.

Hanz · Jul 21, 2016, 1:36 PM

@Tom-Elliott I can confirm location plugin in use on my end

Joe Gill · Jul 21, 2016, 9:11 PM

No new revelations on this front?

I’ll be gone until Monday… I’ve been able to do unicasting but I’d love to get multicasting working because I’m way behind. Thanks!

Joe

Tom Elliott · Jul 21, 2016, 9:15 PM

@Joe-Gill I’m not one who wants to say revert, but I’m told svn version 5887 seems to work okay for Multicast. I imagine/hope this would work for you too.

To install:

svn co -r 5887 https://svn.code.sf.net/p/freeghost/code/trunk
cd trunk/bin
./installfog.sh -y

See if that works, if so, stay there for now.

Joe Gill · Jul 21, 2016, 9:34 PM

@Tom-Elliott

I totally understand. I’ll try that here on Monday. It’s too close to the end of the work day today to do that. Thanks!! I’ll keep you posted!

Tom Elliott · Jul 23, 2016, 4:20 PM

@Joe-Gill I believe I found the issue why your nodes weren’t continually running the service. This was completely unrelated to multicast in and of itself. This should be fixed for RC-2.

For multicast tasks, I believe I’ve found an even keel for things. The location plugin moves hosts to the node to be used. However, if the node was not the “master” node in the past, the host wouldn’t have been looking to the location’s node for its tasking. (This is not specific to you @Joe-Gill rather more to @Hanz and others using location plugin). I’ve created a hook for the FOGMulticastManager service to enable Storage Node Specific locations to start multicast tasks.

All of this should be available fog RC-2 when it get’s published.

Thanks for reporting, and sorry it took so long to respond.

Joe Gill · Jul 25, 2016, 9:01 PM

@Tom-Elliott

Thanks Tom! I have updated to RC-2. I’ll let you know if the problem continues. I’ll be doing more imaging here in the next couple days.

Cheers!

Joe Gill · Jul 27, 2016, 2:44 AM

@Tom-Elliott

Well, I tried running a Multi-task this evening and it hung at the Partclone screen. I did not reset the database prior to trying the Multi-task. I’ll be around tomorrow from
7 AM - 5:30 PM MST. I’m going to update/upgrade to the latest SVN release tonight. But this post is after updating to Running Version 1.3.0-RC-2 SVN Revision: 5937.

Cheers,

Joe

Joe Gill · Jul 27, 2016, 1:23 PM

@Tom-Elliott

Well I managed to get this working again…

Here is what I ended up doing… I upgraded to latest Trunk release (Running Version 1.3.0-RC-4 SVN Revision: 5941) and re-tried doing my multi task… It would not even start a partclone session. The log showed that no tasks were scheduled. It did, however, refresh better this time.

I stopped and started the service on both the Fog Node and the Fog Server. Than I cleared the mysql database. Voila! Things are happy!

I did notice that the multitask log file is logging things better now. I’ll be using this the rest of the week pretty heavily. I’ll keep you posted if things change!

Thanks Tom!!

Joe Gill · Aug 22, 2016, 8:03 PM

@Tom-Elliott

This still seems to haunt me…

Same deal as before… Multicast seems to fail to start. Resetting things and voila, magic. I’m currently on RC-8…

Here is the latest apache2 log file…

root@Fog:/var/log/apache2# tail error.log
[Mon Aug 22 08:28:33.797901 2016] [:error] [pid 1455] [client 172.16.16.150:50548] PHP Warning:  array_map(): Argument #2 should be an array in /var/www/html/fog/lib/pages/dashboardpage.class.php on line 109, referer: http://172.16.1.17/fog/management/index.php
[Mon Aug 22 08:28:33.797930 2016] [:error] [pid 1455] [client 172.16.16.150:50548] PHP Warning:  array_combine() expects parameter 2 to be array, null given in /var/www/html/fog/lib/pages/dashboardpage.class.php on line 110, referer: http://172.16.1.17/fog/management/index.php
[Mon Aug 22 08:28:34.807205 2016] [:error] [pid 1457] [client 172.16.16.150:50563] PHP Warning:  array_map(): Argument #2 should be an array in /var/www/html/fog/lib/pages/dashboardpage.class.php on line 109, referer: http://172.16.1.17/fog/management/index.php
[Mon Aug 22 08:28:34.807232 2016] [:error] [pid 1457] [client 172.16.16.150:50563] PHP Warning:  array_combine() expects parameter 2 to be array, null given in /var/www/html/fog/lib/pages/dashboardpage.class.php on line 110, referer: http://172.16.1.17/fog/management/index.php
[Mon Aug 22 13:27:27.089789 2016] [:error] [pid 1457] [client 172.16.17.122:42012] PHP Warning:  ftp_delete(): Delete operation failed. in /var/www/html/fog/lib/fog/fogftp.class.php on line 59
[Mon Aug 22 13:27:27.090008 2016] [:error] [pid 1457] [client 172.16.17.122:42012] PHP Warning:  ftp_rmdir(): Remove directory operation failed. in /var/www/html/fog/lib/fog/fogftp.class.php on line 173
[Mon Aug 22 13:27:27.103107 2016] [:error] [pid 1457] [client 172.16.17.122:42012] PHP Warning:  ftp_delete(): Delete operation failed. in /var/www/html/fog/lib/fog/fogftp.class.php on line 59
[Mon Aug 22 13:45:30.936479 2016] [:error] [pid 1456] [client 172.16.16.150:61835] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 66, referer: http://172.16.1.17/fog/management/index.php?node=group&sub=edit&id=30
[Mon Aug 22 13:45:48.506431 2016] [:error] [pid 1455] [client 172.16.16.150:61854] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html/fog/lib/fog/group.class.php on line 78, referer: http://172.16.1.17/fog/management/index.php?node=group&sub=edit&id=30
[Mon Aug 22 13:46:53.879293 2016] [:error] [pid 2767] [client 172.16.16.150:61922] PHP Warning:  array_walk() expects parameter 1 to be array, null given in /var/www/html/fog/lib/fog/group.class.php on line 214, referer: http://172.16.1.17/fog/management/index.php?node=group&sub=deploy&id=30&type=8

Particlone Fails to Start

154

12.0k

17.3k

155.2k