Services are not running. Where are the logs?

dolf

I’m trying to debug why my FOG services are not running.

The services exist in the right place, as far as I can see:

root@myserver:/opt/fog/service# ls
etc  FOGImageReplicator  FOGMulticastManager  FOGPingHosts  FOGSnapinReplicator  FOGTaskScheduler  lib

I checked whether they are running:

root@myserver:/etc/init.d# ./FOGImageReplicator status
 * FOGImageReplicator is not running
root@lab2-server:/etc/init.d# ./FOGMulticastManager status
 * FOGMulticastManager is not running
root@lab2-server:/etc/init.d# ./FOGPingHosts status
 * FOGPingHosts is not running
root@lab2-server:/etc/init.d# ./FOGScheduler status
 * FOGScheduler is not running
root@myserver:/etc/init.d# ./FOGSnapinReplicator status
 * FOGSnapinReplicator is not running

They are not. ps confirms this:

# ps aux | grep -i fog
root     20069  0.0  0.0  21096  1016 pts/0    S+   16:20   0:00 grep --color=auto -i fog

Trying to start them manually does not work.

root@myserver:/etc/init.d# ./FOGImageReplicator status
 * FOGImageReplicator is not running
root@myserver:/etc/init.d# ./FOGImageReplicator start
 * Starting FOG Computer Imaging Solution: FOGImageReplicator                                                 [fail]

However, there were .pid files in /var/run/fog! So I deleted them. Now this happens:

root@myserver:/etc/init.d# ./FOGImageReplicator start
 * Starting FOG Computer Imaging Solution: FOGImageReplicator                                                 [ OK ] 
root@myserver:/etc/init.d# ./FOGImageReplicator status
 * FOGImageReplicator is not running

This boggles me. When it [fail]s, it doesn’t give a reason, and when it says [OK] it isn’t, really.

I see that /var/log/fog is a symlink to /opt/fog/log, but that folder is empty:

# ls -lha /opt/fog/log
total 8,0K
drwxr-xr-x 2 root root 4,0K May 16 15:36 .
drwxr-xr-x 6 root root 4,0K Jul  9 11:12 ..

Where can I read about what went wrong?

Scott B

I am also seeing this.

dolf

I forgot to add: Running Fog build 8499 on Ubuntu 14.04.4 LTS

Sebastian Roth

@dolf You might want to temporarily alter the init script and maybe get some more helpful output. The script should look kind of like this. Should be in /etc/init.d/FOGImageReplicator on your system. Open it with an editor and modify line number 31 (could be a different line depending on which version you have installed). Remove the option --quiet from the start-stop-daemon call.

Plus edit /var/www/fog/service/FOGImageReplicator/FOGImageReplicator and change the zero in the third line to one (@error_reporting(1);).

Then run service FOGImageReplicator start and see what you get.

Tom Elliott

I’m going to guess this is HOW you’re running the terminal. You’re getting root how? (Exact steps please).

I’m going to guess (as that’s all I can do at this point) that You’re running root in a non-interactive shell. This means you opened xterm and then obtained root by typing sudo su.

This is incorrect, and I can’t guess this situation nicely.

Try:

sudo -i or sudo su - and see if you can rerun the installer as well as retry the restarting of the services.

Wayne Workman

Remembering back, something like this happened to a fellow once - the cause turned out to be that the execute bit in permissions on the service files was somehow removed…

Tom Elliott

@Wayne-Workman and I’ve updated the installer to correct this.

dolf

tl;dr: Tried everything. No change.

Becomming Jane Root

This is how I become root:

ssh user@myserver
sudo su

Standard procedure, really. This is a non-login shell, but it is not non-interactive. I understand that using -i gives me the root user’s login environment, but IMHO it’s interactive either way. Source.

I originally installed FOG by calling

cd /home/user/fogproject/bin
sudo ./installfog.sh

since that is the “Ubuntu way” of doing things.

@Sebastian-Roth’s advice

Removing --quiet from the start-stop-daemon call does nothing. But it was certainly worth a try.

This is the contents of /var/www/fog/service/:

root@myserver:/var/www/fog/service# ls -lha
total 176K
drwxr-xr-x  3 www-data www-data 4,0K Sep 17  2014 .
drwxr-xr-x 12 www-data www-data 4,0K Sep 17  2014 ..
-rw-r--r--  1 www-data www-data  277 Sep 17  2014 alo-bg.php
-rw-r--r--  1 www-data www-data  739 Sep 17  2014 autologout.php
-rw-r--r--  1 www-data www-data 8,2K Sep 17  2014 auto.register.php
-rw-r--r--  1 www-data www-data  855 Sep 17  2014 av.php
-rw-r--r--  1 www-data www-data 2,4K Sep 17  2014 blame.php
-rw-r--r--  1 www-data www-data 1,4K Sep 17  2014 capone.php
-rw-r--r--  1 www-data www-data  318 Sep 17  2014 checkcredentials.php
-rw-r--r--  1 www-data www-data  108 Sep 17  2014 debug.php
-rw-r--r--  1 www-data www-data  247 Sep 17  2014 dircleanup-dirs.php
-rw-r--r--  1 www-data www-data  979 Sep 17  2014 displaymanager.php
-rw-r--r--  1 www-data www-data  882 Sep 17  2014 donate.php
-rw-r--r--  1 www-data www-data  269 Sep 17  2014 greenfog.php
-rw-r--r--  1 www-data www-data  373 Sep 17  2014 grouplisting.php
-rw-r--r--  1 www-data www-data  402 Sep 17  2014 hostnameloop.php
-rw-r--r--  1 www-data www-data 1013 Sep 17  2014 hostname.php
-rw-r--r--  1 www-data www-data  373 Sep 17  2014 imagelisting.php
-rw-r--r--  1 www-data www-data   51 Sep 17  2014 index.php
-rw-r--r--  1 www-data www-data 4,2K Sep 17  2014 inventory.php
drwxr-xr-x  2 fog      www-data 4,0K Sep 17  2014 ipxe
-rw-r--r--  1 www-data www-data  655 Sep 17  2014 jobs.php
-rw-r--r--  1 www-data www-data  215 Sep 17  2014 locationcheck.php
-rw-r--r--  1 www-data www-data  400 Sep 17  2014 locationlisting.php
-rw-r--r--  1 www-data www-data  719 Sep 17  2014 man.hostexists.php
-rw-r--r--  1 www-data www-data 2,0K Sep 17  2014 mc_checkin.php
-rw-r--r--  1 www-data www-data 3,8K Sep 17  2014 Post_Stage2.php
-rw-r--r--  1 www-data www-data 1,8K Sep 17  2014 Post_Stage3.php
-rw-r--r--  1 www-data www-data  930 Sep 17  2014 Post_Wipe.php
-rw-r--r--  1 www-data www-data 4,2K Sep 17  2014 Pre_Stage1.php
-rw-r--r--  1 www-data www-data 1,5K Sep 17  2014 PrinterManager.php
-rw-r--r--  1 www-data www-data 1,9K Sep 17  2014 progress.php
-rw-r--r--  1 www-data www-data 1,6K Sep 17  2014 register.php
-rw-r--r--  1 www-data www-data 2,0K Sep 17  2014 servicemodule-active.php
-rw-r--r--  1 www-data www-data 1,3K Sep 17  2014 snapcheck.php
-rw-r--r--  1 www-data www-data 3,0K Sep 17  2014 snapins.checkin.php
-rw-r--r--  1 www-data www-data 1,8K Sep 17  2014 snapins.file.php
-rw-r--r--  1 www-data www-data 1,2K Sep 17  2014 updates.php
-rw-r--r--  1 www-data www-data  255 Sep 17  2014 usercleanup-users.php
-rw-r--r--  1 www-data www-data 1,3K Sep 17  2014 usertracking.report.php

But I found FOGImageReplicator and friends in /opt/fog/service. That’s where they should be according to the link you posted (my init.d script looks the same, as expected).

root@myserver:/opt/fog/service# ls -lha
total 36K
drwxr-xr-x 9 root root 4,0K Mar 23 14:43 .
drwxr-xr-x 6 root root 4,0K Jul  9 11:12 ..
drwxr-xr-x 2 root root 4,0K May 15  2014 etc
drwxr-xr-x 2 root root 4,0K May 15  2014 FOGImageReplicator
drwxr-xr-x 2 root root 4,0K May 15  2014 FOGMulticastManager
drwxr-xr-x 2 root root 4,0K Mar 23 14:43 FOGPingHosts
drwxr-xr-x 2 root root 4,0K Mar 23 14:43 FOGSnapinReplicator
drwxr-xr-x 2 root root 4,0K May 15  2014 FOGTaskScheduler
drwxr-xr-x 2 root root 4,0K Mar 23 14:43 lib

Enabling error reporting in FOGImageReplicator also makes no difference Should I be seeing it on the command line, or in a log file somewhere? There are still no files in /opt/fog/log.

@Wayne-Workman’s tip

It looks like the execute bit is set:

root@myserver:/opt/fog/service/FOGImageReplicator# ls -lha
total 12K
drwxr-xr-x 2 root root 4,0K May 15  2014 .
drwxr-xr-x 9 root root 4,0K Mar 23 14:43 ..
-rwxr-xr-x 1 root root  608 Jul 11 22:11 FOGImageReplicator

@Tom-Elliott’s idea

I went into a interactive root login shell with sudo su -. And then:

# service FOGImageReplicator start
 * Starting FOG Computer Imaging Solution: FOGImageReplicator [ OK ] 
# service FOGImageReplicator status
 * FOGImageReplicator is not running

Ran the installer again (still in the shell created with sudo su -

# cd /home/user/fogproject/bin
# ./installfog.sh

Not going to bother you with the entire install log. Every step reported ...OK.

Then lo and behold, still the same

# ls /var/run/fog
FOGImageReplicator.pid  FOGMulticastManager.pid  FOGPingHosts.pid  FOGScheduler.pid  FOGSnapinReplicator.pid
# service FOGImageReplicator status
 * FOGImageReplicator is not running
# service FOGMulticastManager status 
 * FOGMulticastManager is not running
# service FOGPingHosts status
 * FOGPingHosts is not running
# service FOGScheduler status
 * FOGScheduler is not running
# service FOGSnapinReplicator status 
 * FOGSnapinReplicator is not running

Thanks for the help thus far.

Tom Elliott

@dolf I’m trying to reach you. This seems, to me, to be more a need to remote in. I’ve seen this issue come and go and almost every time it’s been sudo issues and/or execute bit.

Tom Elliott

I don’t know how things are running.

Yes it’s normal for people to obtain root sudo su but it’s not giving you the full layer of root’s necessary actions. This presents strangely only on the “debian” based systems that I’ve run into. It’s very tedious and cumbersome to trace issues because you naturally think “I’m already root” but the way things are logged in are not all the same.

yes running sudo ./installfog.sh should work, but I just don’t know all the scenarios this may have issues with.

Sebastian Roth

@dolf Although it seems to not do the trick in your case Tom is right about saying that sudo su does not give you a real proper root shell. While it does switch the user it does not load all the environment (/etc/profile and ~/.bash_profile) which might give you a different behavior in some cases. I have seen issues where starting some of the init services would work but restarting the squid web proxy would just not! Never found out why that is. I suppose it’s the PATH variable not being set properly (/sbin and /usr/sbin missing).

Good you found the init PHP files. Obviously was a bit distracted when I wrote this. What if you run one of the scripts by hand /opt/fog/service/FOGImageReplicator/FOGImageReplicator??

dolf

Eureka! It says:

Missing Extensions: mysqliroot

Sebastian Roth

@dolf Please run this find / -xdev -fname "php.ini*" -exec grep "extension" {} /dev/null \; and post the output here.

dolf

I took the liberty of first:

Correcting the command: find / -xdev -name "php.ini*" -exec grep "extension" {} /dev/null \;
Uninstalling php5 (since I have php5.6 installed, too)

Here is the output: php_extensions.txt

Then I tried this:

# apt-get install php5.6-mysql
# phpenmod mysqli

It runs! BUT there is another error:

PHP Fatal error:  Call to undefined function curl_init() in /var/www/html/fog/lib/fog/fogurlrequests.class.php on line 49

But it didn’t stop running, and a log file appeared at /opt/fog/log!

dolf

And then:

# rm /var/run/fog/*

# service FOGImageReplicator status
 * FOGImageReplicator is not running
# service FOGImageReplicator start
 * Starting FOG Computer Imaging Solution: FOGImageReplicator [ OK ] 
# service FOGImageReplicator status
 * FOGImageReplicator is running

# service FOGMulticastManager status
 * FOGMulticastManager is not running
# service FOGMulticastManager start 
 * Starting FOG Computer Imaging Solution: FOGMulticastManager [ OK ] 
# service FOGMulticastManager status
 * FOGMulticastManager is running

# service FOGPingHosts status
 * FOGPingHosts is not running
# service FOGPingHosts start
 * Starting FOG Computer Imaging Solution: FOGPingHosts [ OK ] 
# service FOGPingHosts status
 * FOGPingHosts is running

# service FOGScheduler status 
 * FOGScheduler is not running
# service FOGScheduler start
 * Starting FOG Computer Imaging Solution: FOGScheduler [ OK ] 
# service FOGScheduler status 
 * FOGScheduler is running

# service FOGSnapinReplicator status
 * FOGSnapinReplicator is not running
# service FOGSnapinReplicator start
 * Starting FOG Computer Imaging Solution: FOGSnapinReplicator [ OK ] 
# service FOGSnapinReplicator status
 * FOGSnapinReplicator is running

But after a reboot, they are not running any more:

# service FOGImageReplicator status
 * FOGImageReplicator is not running
# service FOGMulticastManager status
 * FOGMulticastManager is not running
# service FOGPingHosts status
 * FOGPingHosts is not running
# service FOGScheduler status 
 * FOGScheduler is not running
# service FOGSnapinReplicator status
 * FOGSnapinReplicator is not running

All of the logs in /opt/fog/log contain the same thing:

[07-12-16 9:42:11 am] 
        ___           ___           ___      
       /\  \         /\  \         /\  \     
      /::\  \       /::\  \       /::\  \    
     /:/\:\  \     /:/\:\  \     /:/\:\  \   
    /::\-\:\  \   /:/  \:\  \   /:/  \:\  \  
   /:/\:\ \:\__\ /:/__/ \:\__\ /:/__/_\:\__\ 
   \/__\:\ \/__/ \:\  \ /:/  / \:\  /\ \/__/ 
        \:\__\    \:\  /:/  /   \:\ \:\__\   
         \/__/     \:\/:/  /     \:\/:/  /   
                    \::/  /       \::/  /    
                     \/__/         \/__/     

  ###########################################
  #     Free Computer Imaging Solution      #
  #     Credits:                            #
  #     http://fogproject.org/credits       #
  #     GNU GPL Version 3                   #
  ###########################################

Except the service master:

[07-12-16 9:39:16 am] FOGImageReplicator Start
[07-12-16 9:39:16 am] FOGImageReplicator forked child process (23817).
[07-12-16 9:39:16 am] FOGImageReplicator child process (23817) is running.
[07-12-16 9:39:48 am] service_signal_handler (23816) received signal 2.
[07-12-16 9:39:48 am] service_signal_handler (23816) killing child (23817).
[07-12-16 9:39:48 am] service_signal_handler (23816) exiting.
[07-12-16 9:42:11 am] FOGImageReplicator Start
[07-12-16 9:42:11 am] FOGImageReplicator forked child process (24066).
[07-12-16 9:42:11 am] FOGImageReplicator child process (24066) is running.
[07-12-16 9:42:30 am] FOGMulticastManager Start
[07-12-16 9:42:30 am] FOGMulticastManager forked child process (24139).
[07-12-16 9:42:30 am] FOGMulticastManager child process (24139) is running.
[07-12-16 9:42:42 am] FOGPingHosts Start
[07-12-16 9:42:42 am] FOGPingHosts forked child process (24184).
[07-12-16 9:42:42 am] FOGPingHosts child process (24184) is running.
[07-12-16 9:42:54 am] FOGTaskScheduler Start
[07-12-16 9:42:54 am] FOGTaskScheduler forked child process (24241).
[07-12-16 9:42:54 am] FOGTaskScheduler child process (24241) is running.
[07-12-16 9:43:03 am] FOGImageReplicator Start
[07-12-16 9:43:03 am] FOGImageReplicator forked child process (24300).
[07-12-16 9:43:03 am] FOGImageReplicator child process (24300) is running.
[07-12-16 9:46:11 am] service_signal_handler (24064) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24064) killing child (24066).
[07-12-16 9:46:11 am] service_signal_handler (24064) exiting.
[07-12-16 9:46:11 am] service_signal_handler (24136) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24136) killing child (24139).
[07-12-16 9:46:11 am] service_signal_handler (24136) exiting.
[07-12-16 9:46:11 am] service_signal_handler (24181) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24181) killing child (24184).
[07-12-16 9:46:11 am] service_signal_handler (24181) exiting.
[07-12-16 9:46:11 am] service_signal_handler (24238) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24238) killing child (24241).
[07-12-16 9:46:11 am] service_signal_handler (24238) exiting.
[07-12-16 9:46:11 am] service_signal_handler (24298) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24298) killing child (24300).
[07-12-16 9:46:11 am] service_signal_handler (24298) exiting.

They fail to start using service FOGxxx start, since the pids still exist in /var/run/fog. Deleting those and starting manually as in the beginning of this post works.

Tom Elliott

@dolf said in Services are not running. Where are the logs?:

Call to undefined function curl_init() in

If you run:
sudo apt-get install php5-curl Then:

sudo service FOGMulticastManager stop
sudo service FOGImageReplicator stop
sudo service FOGSnapinReplicator stop
sudo service FOGPingHosts stop
sudo service FOGScheduler stop
sleep 10
sudo service FOGMulticastManager start
sudo service FOGImageReplicator start
sudo service FOGSnapinReplicator start
sudo service FOGPingHosts start
sudo service FOGScheduler start

I think you’re close now.

dolf

My bad. I didn’t enable the apache php5.6 module after uninstalling php5.

# a2enmod php5.6
# service apache2 restart

Still didn’t work. It seems like I broke my php and/or webserver. After reading this, I noticed that the php mime config was missing. So I did the brave thing and purges all versions of php and apache from my server. Starting over…

dolf

AHA! After purging all forms of PHP and Apache from my server, I re-ran the fog installer. (Discovered some apt issues with old and obsolete packages. Killed those by uninstalling everything that I don’t need. Ran the fog installer again. Then it worked.) Still got errors about mysqliroot and curl_init() being missing, but then I had this idea:

Apache2 is using the php5 module, so it always invokes php5. php5-curl and php5-mysql were installed. But the fog installer also caused php7.0 to be installed. php7.0-curl and php7.0-mysql were not installed. So I checked which version was being used on the command line:

# update-alternatives --config php
There are 2 choices for the alternative php (providing /usr/bin/php).

  Selection    Path             Priority   Status
------------------------------------------------------------
* 0            /usr/bin/php7.0   70        auto mode
  1            /usr/bin/php5     50        manual mode
  2            /usr/bin/php7.0   70        manual mode

Press enter to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/bin/php5 to provide /usr/bin/php (php) in manual mode

Sure enough, it started working after changing from php7.0 to php5. All services are running, even after a reboot.

I am a happy bloke.

Sebastian Roth

@dolf Great to see you got this sorted! Seems like the system was a bit messed up.

Wayne Workman

@dolf I guess another route would be to install those php 7 things.

Sounds to me like you didn’t install FOG on Ubuntu 14 with the recommended command for php7. Let me find that…

here’s the command to have FOG use all PHP 7:

php_ver='7.0' php_verAdds='-7.0' ./installfog.sh

If running Ubuntu 16, you’ll need to use that command to start the installer, and THEN when it asks for the DB to be updated - You’d go into MySQL and make a tweak to the root user:
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '';
And then continue to the web interface to update fog.

This is referencing a post I made here:
https://forums.fogproject.org/topic/7131/fog-on-ubuntu-16-04-php7/27

Services are not running. Where are the logs?

Becomming Jane Root

@Sebastian-Roth’s advice

@Wayne-Workman’s tip

@Tom-Elliott’s idea

134

11.7k

17.2k

154.7k