Multicasting stuck on starting to restore image

[arainero]

Hello everyone,

I had a problem in the past that was resolved with Tom’s help, but it has creeped up again. My oringinal post was here. I am using the most recent SVN version - 5319

Almost the exact same symptoms are happening. I can download images to computers with no problem, but when I attempt to multicast 1 or several I get the blue part clone screen and it is stuck at “starting to restore image”.

I did the usual from https://wiki.fogproject.org/wiki//index.php/Troubleshooting_a_multicast:

1. I cleared out the database tables
2. I did the single computer test with success
3. I did the multi-computer test with success

The only relevant log info from /var/log/messages is:

Nov 12 20:55:45 fogserv kernel: hda-intel: IRQ timing workaround is activated for card #0. Suggest a bigger bdl_pos_adj.
Nov 12 20:56:45 fogserv xinetd[1875]: START: tftp pid=2867 from=192.168.1.113
Nov 12 20:56:45 fogserv in.tftpd[2868]: tftp: client does not accept options
Nov 12 20:57:15 fogserv rpc.mountd[1774]: authenticated mount request from 192.168.1.113:837 for /images (/images)

It seems that it gets to reading the image directory and just stops for whatever reason.

I have also tried using a different kernal with no progress.

One thing that I noticed that was weird was if I did a service FOGMulticastManager restart the screen would start to display PHP errors. The only way I could get it to stop was to reboot the machine.

PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14
PHP Fatal error:  Call to undefined method MulticastManager::getBanner() in /opt/fog/service/FOGMulticastManager/FOGMulticastManager on line 14

As always, any help is appreciated. If anyone wants to take a look via teamviewer or anything else, I am more than happy to try.

Thanks!

[Wayne Workman]

Do you have at least one storage node in the storage group you’re using set to be a master node?

[arainero]

I just have the default node and group. I don’t believe I ever modified these settings.

[Wayne Workman]

@arainero Try to set it as the master node.

Also, do you see any messages inside of FOG Configuration -> Log Viewer -> Multicast log ?

[arainero]

The node was already set as master. When I look at the log viewer for multicast I only see

No data to read

Thanks for the help so far.

[Wayne Workman]

@arainero I’ve dug around on the forums and have found a few threads that mention the “no data to read” message…

https://forums.fogproject.org/topic/5951/unable-to-use-log-viewer-from-interface/33?page=1
https://forums.fogproject.org/topic/5504/snap-in-edit-page-not-working
https://forums.fogproject.org/topic/5215/svn-3563-log-viewer-no-data-to-read/5
https://forums.fogproject.org/topic/5988/snapin-unable-to-deploy

It’s likely just a php thing that the @Developers just need to work out, I doubt the message is preventing you from multicasting. However in the above threads, one thing stands out to me… most of them seem to have FTP related issues on the storage node. Can you verify that you can FTP into your server using the credentials listed on your default storage node? There is a walk-through here:
https://wiki.fogproject.org/wiki/index.php/Troubleshoot_FTP

There are messages from the FOGMulticastManager in here: /opt/fog/log/servicemaster.log
Run this command to see the last 20: tail -n 20 /opt/fog/log/servicemaster.log
and run this command to see the entries being made live: tail -f /opt/fog/log/servicemaster.log

It might be worthwhile to see the log live when you try to do a multicast.

Out of curiosity, have you looked at this? https://wiki.fogproject.org/wiki/index.php/Troubleshoot_Downloading_-_Multicast

[arainero]

@Wayne-Workman I think I may be on to the solution. I noticed that that password for storage management was different than everything else. It was just “password”.

However, whenever I change it and update it to match the other passwords it resets back to “password”.

Do you have any thoughts on that?

[Wayne Workman]

@arainero Can you monitor your Apache errors (either through log viewer or through the OS) when you try to update the password in storage management, and see if any errors are generated and post those?

[arainero]

@Wayne-Workman I don’t see anything recent. The most recent error was about 30 minutes ago. I attached it in a text file since the post was getting marked as spam.0_1447388504443_logs.txt

[Wayne Workman]

@arainero Those entries there aren’t about multicast, but the first few are about viewing the logs via the web interface, that will help the Developers figure out what’s going on with that.

Are you still using the same network equipment that you used back when you had SVN 2951 working with multicast? Have you looked through that troubleshooting article I posted?

[arainero]

@Wayne-Workman I am using the same network equipment, nothing has changed. I looked at all of the links. Everything seems to be in order except for the storage managment password not changing from “password”. For testing reason I changed everything else to “password” such as the fog user, tftp, and .fogsettings.

When I attempted to do a multicast then I was given this error"

Failed to create deployment tasking for the following hosts

FOGFTP: Login failed. Host: 192.168.1.3, Username: fog, Password: c588c8, Error: ftp_login(): Login incorrect. 

It attempted to use the old password for some reason.

[arainero]

I think I found a glitch in regards to the password. I think when you change the storage management password it changes on the backend, but when you check it again “password” is still displayed. After going back to storage management and changing “password” to “password” the bad FTP password error went away and the deployment started.

The multicast still failed though, so this wasn’t a password problem and must be somewhere else.

[Tom Elliott]

@arainero I don’t think you found a glitch. I will do what I can to figure out the log reading problems but that is still unlikely the problem. The issue I’m thinking it is is the multicast interface at which its attempting to use. What is your interfaces name? Is it eth0? You can find out by running the commands ifconfig -a or ip addr show. Look for the outputs that show your servers IP address (the one you use to access the GUI) and you should see what it knows for the interface. Then you can look at the node and verify if the interface as presented on the GUI matches the interface your server knows about.

If that still fails it could be the error and I need to figure out why it isn’t creating the log files. That’s relatively simple and you can see what the output says for the service
Master logs.

[arainero]

@Tom-Elliott The interface is eth0 for ifconfig and in the storage node. Here are some recent logs from Service Master:

[11-13-15 10:19:17 pm] FOGMulticastManager fork()ed child process (25996).
[11-13-15 10:19:17 pm] FOGMulticastManager child process (25996) is running.
[11-13-15 10:19:17 pm] FOGMulticastManager fork()ed child process (25997).
[11-13-15 10:19:17 pm] FOGMulticastManager child process (25997) is running.
[11-13-15 10:19:17 pm] FOGImageReplicator fork()ed child process (25998).
[11-13-15 10:19:17 pm] FOGImageReplicator child process (25998) is running.
[11-13-15 10:19:17 pm] FOGImageReplicator fork()ed child process (25999).
[11-13-15 10:19:17 pm] FOGImageReplicator child process (25999) is running.
[11-13-15 10:19:19 pm] FOGImageReplicator fork()ed child process (26001).
[11-13-15 10:19:19 pm] FOGImageReplicator child process (26001) is running.
[11-13-15 10:19:19 pm] FOGTaskScheduler child process (25995) exited with code 2                                                55.
[11-13-15 10:19:20 pm] FOGTaskScheduler fork()ed child process (26002).
[11-13-15 10:19:20 pm] FOGTaskScheduler child process (26002) is running.
[11-13-15 10:19:20 pm] FOGImageReplicator fork()ed child process (26003).
[11-13-15 10:19:20 pm] FOGImageReplicator child process (26003) is running.
[11-13-15 10:19:20 pm] FOGMulticastManager child process (25996) exited with cod                                                e 255.
[11-13-15 10:19:20 pm] FOGMulticastManager child process (25997) exited with cod                                                e 255.
[11-13-15 10:19:20 pm] FOGImageReplicator child process (25998) exited with code                                                 255.
[11-13-15 10:19:20 pm] FOGImageReplicator child process (25999) exited with code                                                 255.
[11-13-15 10:19:22 pm] FOGImageReplicator child process (26001) exited with code                                                 255.
[11-13-15 10:19:22 pm] FOGTaskScheduler fork()ed child process (26004).
[11-13-15 10:19:22 pm] FOGTaskScheduler child process (26004) is running.
[11-13-15 10:19:23 pm] FOGTaskScheduler child process (26002) exited with code 2                                                55.
[11-13-15 10:19:23 pm] FOGImageReplicator child process (26003) exited with code                                                 255.
[11-13-15 10:19:23 pm] FOGMulticastManager fork()ed child process (26005).
[11-13-15 10:19:23 pm] FOGMulticastManager child process (26005) is running.
[11-13-15 10:19:23 pm] FOGMulticastManager fork()ed child process (26006).
[11-13-15 10:19:23 pm] FOGMulticastManager child process (26006) is running.
[11-13-15 10:19:23 pm] FOGImageReplicator fork()ed child process (26007).
[11-13-15 10:19:23 pm] FOGImageReplicator child process (26007) is running.
[11-13-15 10:19:23 pm] FOGImageReplicator fork()ed child process (26008).
[11-13-15 10:19:23 pm] FOGImageReplicator child process (26008) is running.
[11-13-15 10:19:25 pm] FOGImageReplicator fork()ed child process (26009).
[11-13-15 10:19:25 pm] FOGImageReplicator child process (26009) is running.
[11-13-15 10:19:25 pm] FOGTaskScheduler child process (26004) exited with code 255.
[11-13-15 10:19:26 pm] FOGTaskScheduler fork()ed child process (26010).
[11-13-15 10:19:26 pm] FOGTaskScheduler child process (26010) is running.
[11-13-15 10:19:26 pm] FOGImageReplicator fork()ed child process (26011).
[11-13-15 10:19:26 pm] FOGImageReplicator child process (26011) is running.
[11-13-15 10:19:26 pm] FOGMulticastManager child process (26005) exited with code 255.
[11-13-15 10:19:26 pm] FOGMulticastManager child process (26006) exited with code 255.
[11-13-15 10:19:26 pm] FOGImageReplicator child process (26007) exited with code 255.
[11-13-15 10:19:26 pm] FOGImageReplicator child process (26008) exited with code 255.
[11-13-15 10:19:28 pm] FOGImageReplicator child process (26009) exited with code 255.

[Wayne Workman]

@arainero said:

 FOGFTP: Login failed. Host: 192.168.1.3, Username: fog, Password: c588c8, Error: ftp_login(): Login incorrect. 

You’re web interface for updating the password is somehow broken. I don’t know why, but you can fix it manually inside the data base.

mysql
use fog
update globalSettings set settingValue = 'YourNewPassword' where settingKey='FOG_TFTP_FTP_PASSWORD';

Confirm with:
select * from globalSettings where settingKey='FOG_TFTP_FTP_PASSWORD';

See if this helps and please report back.

[Tom Elliott]

@Wayne-Workman this will not fix the issue because the files and data are not read using that password, it’s the information of the storage node that is in use. I will attempt to see if I can replicate the issue as described pertaining to the password updating but @arainero is correct that this is not the issue with multicast or any of the services (besides the replicators maybe) failing.

[Quazz]

@Wayne-Workman

The Storage Node password update thing seems to have trouble whenever you save the password of the WebGUI login (at least it does for chrome, not sure for other browsers)

It will overwrite the password in the field with the WebGUI password.

[Tom Elliott]

@Quazz This sounds like autofill to me.

[Quazz]

@Tom-Elliott You’re correct, but under normal circumstances if you alter the data after it autofills, it will accept and store the new data. It does not for the storage node login information, however.

[Tom Elliott]

@Quazz No, that’s what autofill does Quazz. It tries to guess what you want in that particular field, and the username/password you most commonly use will likely be there.

Now what version of FOG are you running? I’m fairly sure I’ve add a false input field specifically so that it would be prepopulated with the autofill data, so this issue would not occur. Then again, I can’t fix the browser’s coding.