FOGMulticastManager won't start



  • I have two different Ubuntu 14.0.4, Trunk 8088 servers that I can’t get to multicast. I’m working on an additional FOG server in each location so I can do the testing described in “Troubleshoot Downloading - Multicast” wiki article. I have already cleared database as described in article, re-run installer and rebooted several times - no luck. I can’t get FOG MulticastManager service to start, stop, or restart. Here’s last 20 entries in Apache Error logs:

    [Mon Jun 13 11:51:00.859643 2016] [:error] [pid 25884] [client 192.168.1.251:64102] PHP Warning: fopen(/var/log/apache2/error.log): failed to open stream: Permission denied in /var/www/html/fog/status/logtoview.php on line 14, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:00.948499 2016] [:error] [pid 25783] [client 192.168.1.251:64103] PHP Warning: fopen(/var/log/apache2/error.log): failed to open stream: Permission denied in /var/www/html/fog/status/logtoview.php on line 14, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:03.014450 2016] [:error] [pid 25782] [client 192.168.1.251:64110] PHP Warning: fopen(/var/log/apache2/error.log): failed to open stream: Permission denied in /var/www/html/fog/status/logtoview.php on line 14, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:05.316755 2016] [:error] [pid 25785] [client 192.168.1.251:64133] PHP Warning: fopen(/var/log/apache2/error.log): failed to open stream: Permission denied in /var/www/html/fog/status/logtoview.php on line 14, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:08.817558 2016] [:error] [pid 25784] [client 192.168.1.251:64134] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 32, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:08.853842 2016] [:error] [pid 26629] [client 192.168.1.251:64135] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 32, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:10.949570 2016] [:error] [pid 25781] [client 192.168.1.251:64137] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 32, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:10.983563 2016] [:error] [pid 25884] [client 192.168.1.251:64138] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 32, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:11.039406 2016] [:error] [pid 25783] [client 192.168.1.251:64139] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 32, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:11.087299 2016] [:error] [pid 25782] [client 192.168.1.251:64140] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 32, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:13.069976 2016] [:error] [pid 25785] [client 192.168.1.251:64141] PHP Warning: PDO::__construct(): MySQL server has gone away in /var/www/html/fog/lib/db/pdodb.class.php on line 32, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:13.296923 2016] [:error] [pid 25785] [client 192.168.1.251:64141] PHP Warning: fopen(/var/log/apache2/error.log): failed to open stream: Permission denied in /var/www/html/fog/status/logtoview.php on line 14, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:15.424429 2016] [:error] [pid 25784] [client 192.168.1.251:64142] PHP Warning: fopen(/var/log/apache2/error.log): failed to open stream: Permission denied in /var/www/html/fog/status/logtoview.php on line 14, referer: http://192.168.132.19/fog/management/index.php?node=about&sub=logviewer
    [Mon Jun 13 11:51:15.983670 2016] [mpm_prefork:notice] [pid 1156] AH00169: caught SIGTERM, shutting down
    [Mon Jun 13 11:51:33.252823 2016] [mpm_prefork:notice] [pid 29140] AH00163: Apache/2.4.20 (Ubuntu) OpenSSL/1.0.2h configured -- resuming normal operations
    [Mon Jun 13 11:51:33.252889 2016] [core:notice] [pid 29140] AH00094: Command line: '/usr/sbin/apache2'
    [Mon Jun 13 11:52:27.771849 2016] [:error] [pid 29143] [client 192.168.1.251:64191] PHP Strict Standards: Only variables should be passed by reference in /var/www/html/fog/lib/pages/schemaupdaterpage.class.php on line 46, referer: http://192.168.132.19/fog/management/index.php?node=schema
    [Mon Jun 13 11:53:32.257703 2016] [mpm_prefork:notice] [pid 29140] AH00169: caught SIGTERM, shutting down
    [Mon Jun 13 11:53:51.418307 2016] [mpm_prefork:notice] [pid 1186] AH00163: Apache/2.4.20 (Ubuntu) OpenSSL/1.0.2h configured -- resuming normal operations
    [Mon Jun 13 11:53:51.418853 2016] [core:notice] [pid 1186] AH00094: Command line: '/usr/sbin/apache2'
    

    FYI -192.168.1.251 is my machine that I’m using to view the GUI. Seems odd the Multicast log has nothing but very old entries in it.



  • @Wayne-Workman Thanks a ton for you offer to assist!!! I’m going to put these servers right. Need to read up on how environments change in Linux as I’m guessing it has to do with my problem. Live and learn! This should be marked solved I think as we can reasonable guess the problem is hidden in how the server was built/upgraded in the first place.


  • Moderator

    @gwhitfield I’m all for you doing it yourself l. If you hit a bump in the new build, make a new thread and you’ll generally get help fast.



  • @Wayne-Workman I have been working on this as much as possible today and realize that my process was built on old instructions, that there are more recent instructions to do this. Specifically in how I have been logged in as my admin account rather than root (sudo -i) and using sudo to get me by, as well as the location of the installation not being /root/fogproject/. I have cleaned that up and feel pretty confident I can produce a better server. My boss has allocated a bit more time to this so I’m prepared to rebuild these servers. We can skip the TeamViewer session if it’s a hassle for you, I just don’t want to cause you any troubles. If you feel it’s something important to look at I’m all for it though?



  • @Wayne-Workman I can do that. Thank you!


  • Moderator

    @gwhitfield You’re not wasting our time. We help people willingly. If you can wait until about 4 PM central standard, I can offer some help through team viewer.



  • @Wayne-Workman Simply trying to not waste anyone’s time, that’s all. I don’t look forward to re-doing 15 FOG servers :-( but being a linux noob makes troubleshooting super difficult.

    So anyway I just now tried what Tom suggested and it didn’t work. Logged in as my “administrator” account. Entered “sudo -i” and re-ran the installer. Rebooted and logged back in as “administrator”, FOGMulticastManager service is not running. Ran "sudo -i " and tried to restart service and it would not. Tried stopping the service, then starting again and it would not. tried restarting and it would not. I also believe it probably has something to do with dependencies and permissions that got changed somehow by the process I was following to do trunk updates but I have no idea where to begin, just not that Linux savvy.


  • Moderator

    @gwhitfield the senior developer suggested changing to sudo -i and then running the installer. This is simple to do, why not just do it?



  • @Tom-Elliott My best bet may be to simply rebuild these servers and quit everyone chasing this. I have tested saving the hosts and images on the original server to another location then replacing them once the update to trunk is done from 1.2 (where I last snapshotted it). I’m multicasting just fine with this one now. Not knowing what caused it leaves me open to doing it to myself again but with good backups I don’t suppose it’s necessary to get too wrapped up in it unless there’s something we can all learn from it. I’m going to have a couple problem servers available for a while as I’m not going to get to them all in the next several weeks.

    @Wayne-Workman - I have yet to compare the folder rights but will do so tomorrow in HIGH hopes its as simple as changing them to mimic a working installation.

    Thanks to both of you gentlemen!!


  • Senior Developer

    @gwhitfield I say that is correct because, first, the source code of the installfog.sh file is already going to call sudo in the case it is detected that a non-root user started the installer, however there’s not a really feasible way for it to know if the environment of root is sourced properly. It’s most likely safer to place yourself in the root environment directly with sudo -i. I don’t know if this IS the case, but I can be more certain knowing the environment is being used as one would normally expect. Even if it still doesn’t cause help you, at least we have a partial stepping stone to work from.



  • @Tom-Elliott It doesn’t appear to help any, the log files still aren’t being created and the FOGMulticastManager service still isn’t starting. Unfortunately I don’t really understand how to use “sudo” I guess. When in terminal mode (using PuTTY, not GUI) I only append it to a command if I get an “access denied” response to the user “administrator” that I’ve created and I’ve done that a thousand times when upgrading to trunk.

    I’m wondering now if there isn’t something in my upgrading process that left me vulnerable to this issue. I follow these instructions I got off another post:

    1. Log in as Administrator
    2. Run “sudo apt-get autoremove”, enter root password and approve
    3. Reboot server (“sudo reboot now”)
    4. Run “sudo apt-get update”, enter password
    5. Run “sudo apt-get upgrade”
    6. Run “sudo apt-get install subversion” - (Go to Step 9 for any server already at Trunk since svn folder is already created)
    7. Enter “cd ~”
    8. Enter “mkdir svn”
    9. Enter “cd svn”
    10. Enter “sudo svn co https://svn.code.sf.net/p/freeghost/code/trunk
    11. Enter “cd ~/svn/trunk/bin”
    12. Enter “sudo ./installfog.sh”
    13. When prompted, open WEB GUI and update database per the prompt

  • Senior Developer

    @gwhitfield That is absolutely correct.



  • @Tom-Elliott I created a user “Administrator” for all GUI interaction so I’m always logged into the GUI as that user. I don’t really ever use the GUI terminal though I have been using root when connected on my Win10 desktop via "WinSCP" to move or copy files such as the scripts for hardware independent driver installs or copying the drivers themselves. I also use PuTTY regularly to do the trunk upgrades but log in as “administrator” and use “sudo” when necessary. This was the case on 5/18 when every one of the servers I updated appears to have stopped working (as far as logging goes).

    Since I don’t use the GUI terminal I’m understanding that I should still try running installer through PuTTY but use “sudo -i” instead of just “sudo”?

    @Wayne-Workman I did try clearing tables on one server and it didn’t appear to work but I’ll try a couple others since that one server is now currently in the process of rebuilding from a previous snapshot.

    Thank you both for your replies!


  • Senior Developer

    @gwhitfield Just a wild stab here. I’ve seen this issue before in the case of Ubuntu (maybe others?) where you are using GUI interface to interact with a Terminal session of user’s?

    What user are you logged in as on the GUI if you are indeed on a gui? If this is headless, I can try to help further.

    The issue I’ve seen with GUI based terminals and Ubuntu is the load up of the root user (especially if you’re logged into the GUI as ROOT – otherwise it’s mostly user error (not bad, just unknowing) is the root user doesn’t load the profile information properly. This leaves some very strange things in the dark for us.

    If you reopen the terminal, but this time run – sudo -i after getting in, and try rerunning the installer, does this help, hurt, or no difference?

    Speaking of the user-error part, if you are in as a different user (regardless of head or headless environment) and you don’t source the environment correctly, things will go strange. All things will seem to work properly, but permissions of the root user aren’t established as it is still under the user sourcing that logged in.

    For example:

    sudo su root or sudo su is not the same as sudo -i or sudo su -


  • Moderator

    MySQL looks fine. Try to clear out the tables in that “troubleshoot downloading - multicast” article, then restart the FOGMulticastManager service. Steps on this are in that article.

    The log issues look permission related, it says permission denied. Check permissions on the logs out. /opt/fog/log



  • Going to try going back to my snapshot just after server was created to see what’s happening there and try updating that to see if it is working after the update. I can go back and forth between snapshots to troubleshoot if anyone sees something.



  • @Sebastian-Roth I noticed in another forum using “aux” instead of ax gave a little more info so just in case it helps:
    0_1465852227010_upload-36e9d17b-6b03-4098-9831-fb4009e498a1



  • @Sebastian-Roth said in FOGMulticastManager won't start:

    service mysql status

    Sure doesn’t seem like a happy response:
    0_1465851566722_upload-b9b66183-8c97-4033-a4a8-6afa00648633

    edit: well, maybe happy after all since my one working server is giving essentially the same response


  • Developer

    ps ax | grep -e mysql -e mariadb and service mysql status…?



  • @Sebastian-Roth Please forgive my ignorance but how do I determine this? MySQL service is running


Log in to reply
 

418
Online

39.3k
Users

11.0k
Topics

104.6k
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.