Fog System Error : Please help



  • Tried to log into one of our fog servers today, and after entering login info, it just refreshed to the login page.
    Rebooted server, and now all I’m getting is the Database Schema Installer / Updater page.
    I tried backing up the sql db using the (cd ~;mysqldump --allow-keywords -x -v fog > fogbackup.sql), and got the output of : – Connecting to localhost…
    mysqldump: Got error: 2002: Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2) when trying to connect.
    I havnet touched anything else. I have no idea what to do going forward.
    Can someone please assist in the hopes that this can be recovered?
    Systems is Fog 1.2.0 / Ubuntu 14.04.2
    Thanks in advance.


  • Moderator

    @george1421 said:

    I see the majority of the deployments here are ubuntu.

    That’s changing. I strongly push red-hat based distros here because they have been less problematic. I also tend to think that my clearly written instructions for Fedora have swayed the newcomers as well. I’m now working on the CentOS 7 article which is almost a carbon copy of the fedora 21 server article. And then I’ll start on a Fedora 22 server article next lol.

    But seriously, just give me a shove in the right direction for partition resizing and I can probably make it really clear and nice looking.


  • Moderator

    I can see for beginners creating a partitions for the image is a bit daunting depending on the distribution. I’m not sure if there is an easy solution for this. I’m sure I could document it for Centos 6 without much pain, but I see the majority of the deployments here are ubuntu.

    As for the vmdk stuff that is pretty easy, to add a second vmdk and mount that for just the /images. As for expanding the vmdk, I do have to look it up every time I do it, but it is also pretty easy.


  • Moderator

    @george1421 I agree 100%. Originally, I was creating a partition just for the images. But the steps just confused most linux new-comers. I wrote the instructions to be as simple as possible.

    Could you help improve them, and give guidance on expanding the partitions on a 2nd virtual hdd?


  • Moderator

    @Wayne-Workman said:

    @slovisa After freeing up some space, you’d repair mysql. let me see if I can find some commands for that… I’ve done this before at my site when a past bug caused apache logs to grow out of control…

    Just a comment here: standard practice is to move the /home /opt (and a few other) directories to something other than the root partition/filesystem. Having the /images directory in the root partition will lead to the corruption the OP has now. While I’m not shaming here, it would be good practice to move the fog /images folder to something other than the root partition, like /opt/images. In addition recommend during the OS build instructions to put the /opt directory on its own hard drive partition. (on a virtual machine I would even recommend its own vmdk file. The logic here is this is where your deployment images are, if you run out of space in the partition, you can extend the vmdk file then extend the fs to give more room for your images quick and easy.)


  • Moderator

    Try this.
    mysqlcheck --repair --all-databases

    resource used:
    http://dev.mysql.com/doc/refman/5.1/en/rebuilding-tables.html


  • Moderator

    @slovisa After freeing up some space, you’d repair mysql. let me see if I can find some commands for that… I’ve done this before at my site when a past bug caused apache logs to grow out of control…



  • Looks like it.
    So, If I delete an unused image, to free up some space, what would be my next move?

    it@et1fog01:/etc/init.d$ df -h
    df: ‘/run/user/112/gvfs’: Permission denied
    
    
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sda1       143G  143G     0 100% /
    none            4.0K     0  4.0K   0% /sys/fs/cgroup
    udev            2.0G  4.0K  2.0G   1% /dev
    tmpfs           398M  1.1M  397M   1% /run
    none            5.0M     0  5.0M   0% /run/lock
    none            2.0G  144K  2.0G   1% /run/shm
    none            100M   12K  100M   1% /run/user
    overflow        1.0M  4.0K 1020K   1% /tmp
    

    Mod edited to use code box.


  • Moderator

    @slovisa
    Sounds like one of your partitions is full.
    post the output of df -h



  • @george1421 said:

    sudo /etc/init.d/mysql start

    Here is the output.

    • /etc/init.d/mysql: ERROR: The partition with /var/lib/mysql is too full!

    So…should I just delete an image from the file system?


  • Moderator

    OK on the system history. Sometimes you don’t know unless you ask first.

    As for the commands it appears that the mysql service is not running and since I’m from rhel land (redhat) I don’t know ubuntu commands very well.

    A quick google-fu excersion shows me this command.
    sudo /etc/init.d/mysql start

    That should start the mysql service.



  • The system has been in production for over a year, with no issues noted.
    No changes have been made, besides me logging in today, nobody has logged into it since March.
    Here is me trying the various commands that you suggested.

    it@et1fog01:~$ ps -aux | grep mysql
    it        5716  0.0  0.0   4684  2152 pts/11   S+   12:02   0:00 grep --color=auto mysql
    it@et1fog01:~$ ps -aux | grep mysql
    it        5734  0.0  0.0   4684  2080 pts/11   S+   12:02   0:00 grep --color=auto mysql
    it@et1fog01:~$ it        5716  0.0  0.0   4684  2152 pts/11   S+   12:02   0:00 grep --color=auto mysql
    it: command not found
    it@et1fog01:~$ service mysqld start
    mysqld: unrecognized service
    it@et1fog01:~$ ./mysqld start
    -bash: ./mysqld: No such file or directory
    it@et1fog01:~$ cd /etc/init.d
    it@et1fog01:/etc/init.d$ ./mysqld start
    -bash: ./mysqld: No such file or directory
    it@et1fog01:/etc/init.d$
    

    Mod edited to use code box.


  • Moderator

    Well the error is telling me that the mysql service is not running.

    Tell us a bit more about this system:
    Is this a new install? If not how long has it been in production.?
    Have you made any changes to the server since it last worked?

    On to the issue if you open an command window and issue the following command.

    ps -aux | grep mysql

    Does it show the mysqld service running?

    If not then issue the following command (sorry its in rhel syntax)

    service mysqld start
    

    or change to the /etc/init.d folder and issue

    ./mysqld start
    

    To start the service will be distribution specific but the idea is to start the mysql server and see if you get the same results.


Log in to reply
 

365
Online

38982
Users

10712
Topics

101679
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.