Fog System Error : Please help

[slovisa]

Tried to log into one of our fog servers today, and after entering login info, it just refreshed to the login page.
Rebooted server, and now all I’m getting is the Database Schema Installer / Updater page.
I tried backing up the sql db using the (cd ~;mysqldump --allow-keywords -x -v fog > fogbackup.sql), and got the output of : – Connecting to localhost…
mysqldump: Got error: 2002: Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2) when trying to connect.
I havnet touched anything else. I have no idea what to do going forward.
Can someone please assist in the hopes that this can be recovered?
Systems is Fog 1.2.0 / Ubuntu 14.04.2
Thanks in advance.

[george1421]

Well the error is telling me that the mysql service is not running.

Tell us a bit more about this system:
Is this a new install? If not how long has it been in production.?
Have you made any changes to the server since it last worked?

On to the issue if you open an command window and issue the following command.

ps -aux | grep mysql

Does it show the mysqld service running?

If not then issue the following command (sorry its in rhel syntax)

service mysqld start

or change to the /etc/init.d folder and issue

./mysqld start

To start the service will be distribution specific but the idea is to start the mysql server and see if you get the same results.

[slovisa]

The system has been in production for over a year, with no issues noted.
No changes have been made, besides me logging in today, nobody has logged into it since March.
Here is me trying the various commands that you suggested.

it@et1fog01:~$ ps -aux | grep mysql
it        5716  0.0  0.0   4684  2152 pts/11   S+   12:02   0:00 grep --color=auto mysql
it@et1fog01:~$ ps -aux | grep mysql
it        5734  0.0  0.0   4684  2080 pts/11   S+   12:02   0:00 grep --color=auto mysql
it@et1fog01:~$ it        5716  0.0  0.0   4684  2152 pts/11   S+   12:02   0:00 grep --color=auto mysql
it: command not found
it@et1fog01:~$ service mysqld start
mysqld: unrecognized service
it@et1fog01:~$ ./mysqld start
-bash: ./mysqld: No such file or directory
it@et1fog01:~$ cd /etc/init.d
it@et1fog01:/etc/init.d$ ./mysqld start
-bash: ./mysqld: No such file or directory
it@et1fog01:/etc/init.d$

Mod edited to use code box.

[george1421]

OK on the system history. Sometimes you don’t know unless you ask first.

As for the commands it appears that the mysql service is not running and since I’m from rhel land (redhat) I don’t know ubuntu commands very well.

A quick google-fu excersion shows me this command.
sudo /etc/init.d/mysql start

That should start the mysql service.

[slovisa]

@george1421 said:

sudo /etc/init.d/mysql start

Here is the output.

• /etc/init.d/mysql: ERROR: The partition with /var/lib/mysql is too full!

So…should I just delete an image from the file system?

[Wayne Workman]

@slovisa
Sounds like one of your partitions is full.
post the output of df -h

[slovisa]

Looks like it.
So, If I delete an unused image, to free up some space, what would be my next move?

it@et1fog01:/etc/init.d$ df -h
df: ‘/run/user/112/gvfs’: Permission denied


Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       143G  143G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            2.0G  4.0K  2.0G   1% /dev
tmpfs           398M  1.1M  397M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G  144K  2.0G   1% /run/shm
none            100M   12K  100M   1% /run/user
overflow        1.0M  4.0K 1020K   1% /tmp

Mod edited to use code box.

[Wayne Workman]

@slovisa After freeing up some space, you’d repair mysql. let me see if I can find some commands for that… I’ve done this before at my site when a past bug caused apache logs to grow out of control…

[Wayne Workman]

Try this.
mysqlcheck --repair --all-databases

resource used:
http://dev.mysql.com/doc/refman/5.1/en/rebuilding-tables.html

[george1421]

@Wayne-Workman said:

@slovisa After freeing up some space, you’d repair mysql. let me see if I can find some commands for that… I’ve done this before at my site when a past bug caused apache logs to grow out of control…

Just a comment here: standard practice is to move the /home /opt (and a few other) directories to something other than the root partition/filesystem. Having the /images directory in the root partition will lead to the corruption the OP has now. While I’m not shaming here, it would be good practice to move the fog /images folder to something other than the root partition, like /opt/images. In addition recommend during the OS build instructions to put the /opt directory on its own hard drive partition. (on a virtual machine I would even recommend its own vmdk file. The logic here is this is where your deployment images are, if you run out of space in the partition, you can extend the vmdk file then extend the fs to give more room for your images quick and easy.)

[Wayne Workman]

@george1421 I agree 100%. Originally, I was creating a partition just for the images. But the steps just confused most linux new-comers. I wrote the instructions to be as simple as possible.

Could you help improve them, and give guidance on expanding the partitions on a 2nd virtual hdd?

[george1421]

I can see for beginners creating a partitions for the image is a bit daunting depending on the distribution. I’m not sure if there is an easy solution for this. I’m sure I could document it for Centos 6 without much pain, but I see the majority of the deployments here are ubuntu.

As for the vmdk stuff that is pretty easy, to add a second vmdk and mount that for just the /images. As for expanding the vmdk, I do have to look it up every time I do it, but it is also pretty easy.

[Wayne Workman]

@george1421 said:

I see the majority of the deployments here are ubuntu.

That’s changing. I strongly push red-hat based distros here because they have been less problematic. I also tend to think that my clearly written instructions for Fedora have swayed the newcomers as well. I’m now working on the CentOS 7 article which is almost a carbon copy of the fedora 21 server article. And then I’ll start on a Fedora 22 server article next lol.

But seriously, just give me a shove in the right direction for partition resizing and I can probably make it really clear and nice looking.