FOGImagereplicator not working

[Neil Underwood]

For some time now my images have not been replicating and I’ve had to transfer all new images manually via scp. I’m not sure at what point this happened, but it was after an update. Additionally, there is absolutely nothing in the image replicator log. The logging stopped at some point back in March. All the other logs seem to be working fine though. How would I go about troubleshooting this?

This installation started out on version 1.2.0 over a year ago and I have been periodically upgrading through git since. At no point have I done a fresh installation. Should I have done that at some point? And, if I do, should I be able to migrate the database over fairly easily between versions?

[Wayne Workman]

@Neil-Underwood You should not need to do a re-install, we can probably fix it.

This is 99% of the time a FTP issue, 95% of the time a credentials issue, and 50% of the time caused by your web-browser’s auto-fill.

This is a troubleshooting guide for all of that:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP

[Neil Underwood]

I haven’t finished trying everything in the link you provided, but I did notice that all my passwords are correct. I also noticed that I can reach each node via ftp://xxx.xxx.xxx.xxx in a web browser using the credentials in the storage settings. Question though: Shouldn’t I be able to go straight to the images directory by entering:
ftp://xxx.xxx.xxx.xxx/images
into the web browser? I have to manually navigate up two levels to get to my root directory , then choose the images folder. Only then do I see the images.

Initially the default ftp directory is empty. The first time I click to navigate up one directory I end up in /home. The second time it takes me to / as expected. I believe it’s starting me out in /home/fog, because that’s the only empty directory in /home. Is this normal?

The strangest thing is I have no problems on the web interface with the network graph and the disk space usage.

[Wayne Workman]

@Neil-Underwood The true test is to connect via CLI, as the tests show in that article, using the credentials you have in storage management.

Also, this is a thread about replication, so you have two or mode nodes. You’re trying to connect to the problematic one correct?

[Neil Underwood]

Well, I have 9 nodes actually, plus the master storage node. None of them are replicating at all. They all give me the same results.

Not sure if it’s related or worth mentioning, but at the same time this started happening, I started having issues with bzImage reaching 99% and then just sitting there when trying to run a task.

[Neil Underwood]

FTP testing checks out with no issues. I can put, get and delete just fine.

[Wayne Workman]

@Neil-Underwood I think this is a problem with the central server. Not sure exactly yet, we need some more information. What OS is the main server? Have you reviewed firewall and SELinux (if applicable)?

For troubleshooting, please just turn off all firewalls on the nodes and main server.

Also, inside of Web Interface -> FOG Configuration -> Log Viewer -> Apache Error Logs What do you see in there, if anything? If that’s blank, get them manually from the OS. Depending on the OS, they are in different spots. RHEL and variants are /var/log/httpd/error_log and Debian variants are /var/log/apache2/error_log

And, you can find replication logs on the main server here: /opt/fog/log

[Neil Underwood]

I’m on Ubuntu 14.04. No firewall, no SElinux. In the web interface I get “Unable to open file for reading” for all apache logs. When viewing them manually I get:

[Mon Jun 20 07:35:03.417754 2016] [mpm_prefork:notice] [pid 1761] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2h configured -- resuming normal operations
[Mon Jun 20 07:35:03.417771 2016] [core:notice] [pid 1761] AH00094: Command line: '/usr/sbin/apache2'
[Mon Jun 20 14:39:59.444514 2016] [:error] [pid 14709] [client 192.168.18.151:38917] PHP Warning:  fopen(/var/log/apache2/error.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:03.291699 2016] [:error] [pid 14419] [client 192.168.18.151:38918] PHP Warning:  fopen(/var/log/apache2/access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:04.386138 2016] [:error] [pid 14420] [client 192.168.18.151:38921] PHP Warning:  fopen(/var/log/apache2/access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:06.568418 2016] [:error] [pid 14417] [client 192.168.18.151:38923] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:09.496948 2016] [:error] [pid 14419] [client 192.168.18.151:38924] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:13.349986 2016] [:error] [pid 14455] [client 192.168.18.151:38926] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:14.435769 2016] [:error] [pid 14418] [client 192.168.18.151:38928] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:16.612700 2016] [:error] [pid 14709] [client 192.168.18.151:38929] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:19.541410 2016] [:error] [pid 14418] [client 192.168.18.151:38930] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:23.393842 2016] [:error] [pid 14421] [client 192.168.18.151:38931] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:24.482583 2016] [:error] [pid 14709] [client 192.168.18.151:38933] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:26.658868 2016] [:error] [pid 14456] [client 192.168.18.151:38934] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:29.587489 2016] [:error] [pid 14455] [client 192.168.18.151:38935] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:33.436966 2016] [:error] [pid 14709] [client 192.168.18.151:38936] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:34.529403 2016] [:error] [pid 14418] [client 192.168.18.151:38938] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:36.700967 2016] [:error] [pid 14419] [client 192.168.18.151:38939] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:39.632103 2016] [:error] [pid 17625] [client 192.168.18.151:38940] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:43.481757 2016] [:error] [pid 14456] [client 192.168.18.151:38941] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:44.561740 2016] [:error] [pid 14527] [client 192.168.18.151:38943] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:46.744553 2016] [:error] [pid 14420] [client 192.168.18.151:38944] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:49.667644 2016] [:error] [pid 14456] [client 192.168.18.151:38945] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:53.536456 2016] [:error] [pid 17625] [client 192.168.18.151:38948] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:54.604695 2016] [:error] [pid 14709] [client 192.168.18.151:38950] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:56.789575 2016] [:error] [pid 14527] [client 192.168.18.151:38951] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:40:59.708623 2016] [:error] [pid 14527] [client 192.168.18.151:38952] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:41:03.581499 2016] [:error] [pid 14419] [client 192.168.18.151:38955] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:41:04.648658 2016] [:error] [pid 14527] [client 192.168.18.151:38956] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:41:06.832612 2016] [:error] [pid 17625] [client 192.168.18.151:38957] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:41:09.749289 2016] [:error] [pid 14420] [client 192.168.18.151:38958] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:41:13.625475 2016] [:error] [pid 14456] [client 192.168.18.151:38960] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer
[Mon Jun 20 14:41:14.692155 2016] [:error] [pid 17625] [client 192.168.18.151:38961] PHP Warning:  fopen(/var/log/apache2/other_vhosts_access.log): failed to open stream: Permission denied in /var/www/fog/status/logtoview.php on line 14, referer: http://192.168.19.200/fog/management/index.php?node=about&sub=logviewer

The fogreplicator.log file is completely empty. I deleted it and recreated it a few weeks ago during my troubleshooting. Prior to that it contained entries up to March and then it stopped.

Permissions in /var/log/apache2 are:

sradmin@new-fog-server ~ $ ll /var/log/apache2/*
-rw-r----- 1 root adm        0 Jun 17 07:35 /var/log/apache2/access.log
-rw-r----- 1 root adm      414 Jun 16 15:45 /var/log/apache2/access.log.1
-rwxr-xr-x 1 root adm      147 May  9 06:01 /var/log/apache2/access.log.10.gz
-rwxr-xr-x 1 root adm      167 May  7 07:55 /var/log/apache2/access.log.11.gz
-rwxr-xr-x 1 root adm      718 May  7 02:15 /var/log/apache2/access.log.12.gz
-rwxr-xr-x 1 root adm      287 May  6 07:04 /var/log/apache2/access.log.13.gz
-rwxr-xr-x 1 root adm      336 May  5 06:05 /var/log/apache2/access.log.14.gz
-rw-r----- 1 root adm      165 Jun 15 11:16 /var/log/apache2/access.log.2.gz
-rw-r----- 1 root adm      210 Jun 13 15:42 /var/log/apache2/access.log.3.gz
-rw-r----- 1 root adm      101 Jun 10 08:03 /var/log/apache2/access.log.4.gz
-rwxr-xr-x 1 root adm      252 Jun  9 22:58 /var/log/apache2/access.log.5.gz
-rwxr-xr-x 1 root adm      423 Jun  9 11:27 /var/log/apache2/access.log.6.gz
-rwxr-xr-x 1 root adm      171 May 25 15:30 /var/log/apache2/access.log.7.gz
-rwxr-xr-x 1 root adm      147 May 23 02:06 /var/log/apache2/access.log.8.gz
-rwxr-xr-x 1 root adm      147 May 17 14:23 /var/log/apache2/access.log.9.gz
-rw-r----- 1 root adm    74862 Jun 20 14:49 /var/log/apache2/error.log
-rw-r----- 1 root adm      373 Jun 20 07:35 /var/log/apache2/error.log.1
-rw-r----- 1 root adm      252 Jun 11 07:35 /var/log/apache2/error.log.10.gz
-rwxr-xr-x 1 root adm     2720 Jun 10 06:34 /var/log/apache2/error.log.11.gz
-rwxr-xr-x 1 root adm    62439 Jun  9 11:32 /var/log/apache2/error.log.12.gz
-rwxr-xr-x 1 root adm      249 May 31 07:35 /var/log/apache2/error.log.13.gz
-rwxr-xr-x 1 root adm      249 May 30 07:35 /var/log/apache2/error.log.14.gz
-rw-r----- 1 root adm      247 Jun 19 07:35 /var/log/apache2/error.log.2.gz
-rw-r----- 1 root adm      250 Jun 18 07:35 /var/log/apache2/error.log.3.gz
-rw-r----- 1 root adm      272 Jun 17 07:35 /var/log/apache2/error.log.4.gz
-rw-r----- 1 root adm      271 Jun 16 07:35 /var/log/apache2/error.log.5.gz
-rw-r----- 1 root adm      702 Jun 15 07:35 /var/log/apache2/error.log.6.gz
-rw-r----- 1 root adm      463 Jun 14 07:35 /var/log/apache2/error.log.7.gz
-rw-r----- 1 root adm      247 Jun 13 07:35 /var/log/apache2/error.log.8.gz
-rw-r----- 1 root adm      246 Jun 12 07:35 /var/log/apache2/error.log.9.gz
-rw-r----- 1 root adm 15458893 Jun 20 14:49 /var/log/apache2/other_vhosts_access.log
-rw-r----- 1 root adm 37466046 Jun 20 07:35 /var/log/apache2/other_vhosts_access.log.1
-rw-r----- 1 root adm  1439995 Jun 11 07:35 /var/log/apache2/other_vhosts_access.log.10.gz
-rwxr-xr-x 1 root adm   990971 Jun 10 06:34 /var/log/apache2/other_vhosts_access.log.11.gz
-rwxr-xr-x 1 root adm 30755510 Jun  9 11:32 /var/log/apache2/other_vhosts_access.log.12.gz
-rwxr-xr-x 1 root adm  3834221 May 31 07:35 /var/log/apache2/other_vhosts_access.log.13.gz
-rwxr-xr-x 1 root adm  3817897 May 30 07:35 /var/log/apache2/other_vhosts_access.log.14.gz
-rw-r----- 1 root adm  1923689 Jun 19 07:35 /var/log/apache2/other_vhosts_access.log.2.gz
-rw-r----- 1 root adm  2099730 Jun 18 07:35 /var/log/apache2/other_vhosts_access.log.3.gz
-rw-r----- 1 root adm  2117286 Jun 17 07:35 /var/log/apache2/other_vhosts_access.log.4.gz
-rw-r----- 1 root adm  1921642 Jun 16 07:35 /var/log/apache2/other_vhosts_access.log.5.gz
-rw-r----- 1 root adm  1904000 Jun 15 07:35 /var/log/apache2/other_vhosts_access.log.6.gz
-rw-r----- 1 root adm  1886950 Jun 14 07:35 /var/log/apache2/other_vhosts_access.log.7.gz
-rw-r----- 1 root adm  1449643 Jun 13 07:35 /var/log/apache2/other_vhosts_access.log.8.gz
-rw-r----- 1 root adm  1442578 Jun 12 07:35 /var/log/apache2/other_vhosts_access.log.9.gz

sradmin@new-fog-server ~ $ ll /opt/fog/log/*
-rwxrwxrwx 1 fog root       0 Jun  6 11:43 /opt/fog/log/fogreplicator.log
-rwxrwxrwx 1 fog root 3500683 Apr  4 09:28 /opt/fog/log/fogscheduler.log
-rwxrwxrwx 1 fog root  964464 Mar  4 12:03 /opt/fog/log/fogsnapinrep.log
-rwxrwxrwx 1 fog root  355229 Mar  9 01:39 /opt/fog/log/groupmanager.log
-rwxrwxrwx 1 fog root 8503470 Apr  4 09:29 /opt/fog/log/multicast.log
-rwxrwxrwx 1 fog root     139 Aug 25  2015 /opt/fog/log/multicast.log.udpcast.4
-rwxrwxrwx 1 fog root   95930 Aug 25  2015 /opt/fog/log/multicast.log.udpcast.5
-rwxrwxrwx 1 fog root  346282 Apr  4 09:27 /opt/fog/log/pinghosts.log
-rwxrwxrwx 1 fog root 7270442 Jun 20 14:49 /opt/fog/log/servicemaster.log

sradmin@new-fog-server ~ $ ll /var/log/fog/*
-rwxrwxrwx 1 fog root       0 Jun  6 11:43 /var/log/fog/fogreplicator.log
-rwxrwxrwx 1 fog root 3500683 Apr  4 09:28 /var/log/fog/fogscheduler.log
-rwxrwxrwx 1 fog root  964464 Mar  4 12:03 /var/log/fog/fogsnapinrep.log
-rwxrwxrwx 1 fog root  355229 Mar  9 01:39 /var/log/fog/groupmanager.log
-rwxrwxrwx 1 fog root 8503470 Apr  4 09:29 /var/log/fog/multicast.log
-rwxrwxrwx 1 fog root     139 Aug 25  2015 /var/log/fog/multicast.log.udpcast.4
-rwxrwxrwx 1 fog root   95930 Aug 25  2015 /var/log/fog/multicast.log.udpcast.5
-rwxrwxrwx 1 fog root  346282 Apr  4 09:27 /var/log/fog/pinghosts.log
-rwxrwxrwx 1 fog root 7270442 Jun 20 14:49 /var/log/fog/servicemaster.log

/opt/fog/log/* permissions are still 777 from my previous troubleshooting attempts, as are /var/log/fog/*

[Tom Elliott]

@Neil-Underwood Please give this a shot:

sudo -i
service FOGImageReplicator stop
service FOGSnapinReplicator stop
service FOGMulticastManager stop
service FOGPingHosts stop
service FOGScheduler stop
sleep 10
service FOGImageReplicator starts
service FOGSnapinReplicator start
service FOGMulticastManager start
service FOGPingHosts start
service FOGScheduler start

Ubuntu is realy strange. If there’s a pid file, but the process died, it will fail to allow the next item to startup (at least not properly). Restart appears to make it not remove the lock, at least in my experiences.

Also, if you’re running from a GUI 14.04, make sure when you go into sudo on the terminal, you first jump up to root with sudo -i. The normal terminal window (even if you are logged in to the Ubuntu system as root) will not source all the paths as needed. This is what I’ve seen, of course I have been wrong.

[Wayne Workman]

@Tom-Elliott said in FOGImagereplicator not working:

@Neil-Underwood Please give this a shot:

sudo -i
service FOGImageReplicator stop
service FOGSnapinReplicator stop
service FOGMulticastManager stop
service FOGPingHosts stop
service FOGScheduler stop
sleep 10
service FOGImageReplicator starts
service FOGSnapinReplicator start
service FOGMulticastManager start
service FOGPingHosts start
service FOGScheduler start

Ubuntu is realy strange. If there’s a pid file, but the process died, it will fail to allow the next item to startup (at least not properly). Restart appears to make it not remove the lock, at least in my experiences.

Also, if you’re running from a GUI 14.04, make sure when you go into sudo on the terminal, you first jump up to root with sudo -i. The normal terminal window (even if you are logged in to the Ubuntu system as root) will not source all the paths as needed. This is what I’ve seen, of course I have been wrong.

#wiki worthy

[Neil Underwood]

@Tom-Elliott said in FOGImagereplicator not working:

@Neil-Underwood Please give this a shot:

sudo -i
service FOGImageReplicator stop
service FOGSnapinReplicator stop
service FOGMulticastManager stop
service FOGPingHosts stop
service FOGScheduler stop
sleep 10
service FOGImageReplicator starts
service FOGSnapinReplicator start
service FOGMulticastManager start
service FOGPingHosts start
service FOGScheduler start

Ubuntu is realy strange. If there’s a pid file, but the process died, it will fail to allow the next item to startup (at least not properly). Restart appears to make it not remove the lock, at least in my experiences.

Also, if you’re running from a GUI 14.04, make sure when you go into sudo on the terminal, you first jump up to root with sudo -i. The normal terminal window (even if you are logged in to the Ubuntu system as root) will not source all the paths as needed. This is what I’ve seen, of course I have been wrong.

I tried this with no success. Did it twice to make sure. The first time I stopped FOGImageReplicator and FOGSnapinReplicator they both complained about no existing process, but agreed to close anyway. The second time I did it they did not complain but the outcome was the same.

Looking more closely at all my log files, I see that only servicemaster.log is current. Everything else hasn’t changed since March/April. Looking at multicast.log I see something that looks very wrong:

[04-04-16 9:08:19 am]  * No tasks found!
[04-04-16 9:08:29 am]  * No tasks found!
[04-04-16 9:08:39 am]  * No tasks found!
[04-04-16 9:08:49 am]  * No tasks found!
[04-04-16 9:08:59 am]  * No tasks found!
[04-04-16 9:09:09 am]  * No tasks found!
[04-04-16 9:09:19 am]  | This is not the master node
[04-04-16 9:09:29 am]  | This is not the master node
[04-04-16 9:09:39 am]  | This is not the master node
[04-04-16 9:09:49 am]  | This is not the master node
[04-04-16 9:09:59 am]  | This is not the master node
[04-04-16 9:10:09 am]  | This is not the master node
[04-04-16 9:10:19 am]  | This is not the master node
[04-04-16 9:10:29 am]  | This is not the master node
[04-04-16 9:10:39 am]  | This is not the master node
[04-04-16 9:10:49 am]  | This is not the master node
[04-04-16 9:10:59 am]  | This is not the master node
[04-04-16 9:11:09 am]  | This is not the master node
[04-04-16 9:11:19 am]  | This is not the master node
[04-04-16 9:11:29 am]  | This is not the master node
[04-04-16 9:11:39 am]  | This is not the master node
[04-04-16 9:11:49 am]  | This is not the master node
[04-04-16 9:11:59 am]  | This is not the master node
[04-04-16 9:12:09 am]  | This is not the master node
[04-04-16 9:12:19 am]  | This is not the master node
[04-04-16 9:12:29 am]  | This is not the master node
[04-04-16 9:12:39 am]  | This is not the master node
[04-04-16 9:12:49 am]  | This is not the master node

Not sure how this box lost master node status. It’s never been changed. That is a bit worrisome though. I feel like there are multiple issues with this setup.

[Neil Underwood]

Here’s another side-effect of whatever is ailing my server here:

All of the nodes now show (#!db) instead of whatever it’s supposed to show (version number I think?)
Could it be a corrupt database that’s causing all of this?

[Wayne Workman]

Looking at that picture, this issue is possibly related: https://forums.fogproject.org/topic/7825/db/17

[Neil Underwood]

@Wayne-Workman said in FOGImagereplicator not working:

Looking at that picture, this issue is possibly related: https://forums.fogproject.org/topic/7825/db/17

You know, that prompted me to look at class.config.php and I think the DATABASE_HOST value is wrong, but I really don’t know. Can you verify this is correct? Should the “p:” be in there proceeding the IP address?

define('DATABASE_HOST','p:127.0.0.1');

I tried changing it and removing the “p:” and nothing seemed to change (after restarting php, mysql & apache just to be sure)

[Wayne Workman]

@Neil-Underwood so, for the main server that hosts the DB, that is correct. For all nodes, it is wrong.

[Neil Underwood]

OK, so now I’m seeing that I can’t connect to the master storage node DB from the remote storage nodes. I know as much about SQL commands and permissions as my left shoe. How do I fix this? And how in the heck did it break in the first place?

[Wayne Workman]

@Neil-Underwood said in FOGImagereplicator not working:

w do I fix this? And how in the heck did it break in the first place?

Probably the server is only allowing local connections. There’s stuff about that in here:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_MySQL

“Enable Remote MySQL Access”

[Neil Underwood]

@Wayne-Workman said in FOGImagereplicator not working:

@Neil-Underwood said in FOGImagereplicator not working:

w do I fix this? And how in the heck did it break in the first place?

Probably the server is only allowing local connections. There’s stuff about that in here:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_MySQL

“Enable Remote MySQL Access”

I must have overlooked the part about remote access because I visited this page earlier this morning. I fixed the remote permissions and now everything is able to see the DB. Thanks for that. I did do an OS upgrade that probably changed mysql versions. I’m betting that’s what screwed it up.

The mystery of where my log info is going still remains, however.

Also, on the dashboard the replicator appears to be working, based on the graph. But running jnettop from the master storage node doesn’t show any replication activity. If I can get my logs working again I’m certain I can figure out what the problem is.

[Wayne Workman]

@Neil-Underwood sometimes it takes a hot second for replication to kick in, even if the log says it’s going. Just be patient. Tomorrow morning you’ll be able to tell if everything is right or not.

Also, the logs might be fixed by re-running the installer on all the nodes, since it can now actually communicate with the DB.

[Neil Underwood]

OK so I’m updated to 8215 now and everything seems to be able to see each other. It looks like the main server is pushing our the smartinstaller.exe to all the clients and that’s the activity I’ve been seeing on the graph. Hard to catch that using jnettop at first.

Thanks for all the assistance with this. I’ll open a separate thread for the log issue and another for the bzImage problem.

One more quick question though. What are these values used for?

startrange='192.168.19.10'
endrange='192.168.19.254'