Database Connection Unavailable
-
Hi all,
Attempted to log onto my FOG server yesterday afternoon and got a ‘database connection unavailable error’. I did some reading on the forum and came across this thread - https://forums.fogproject.org/topic/10006/ubuntu-is-fog-s-enemy which seems to be the issue I’m having in that it’s on Ubuntu 16.
I tried the first suggestion of re-running the fog server by doing the following -
- sudo -i
- cd /root/fogproject
- git checkout dev-branch
- git pull
- cd bin
- ./installfog.sh
This completed however I’m still getting the same error and I can see in the install that mysql part completes -
I then tried the second suggestion there which was opening MYSQL but when I do that, it’s unable to connect to it and I then try start the service and that has an error which I could see in the log -
I tried to roll this back to a veeam image from a couple of weeks ago and I’m getting the same issue. It might be that I need to go back further but I think I read on here that others who have tried the same had the same issue of it not fixing the database.
I’m hoping I’ve missed something simple in what I should be doing but I’m just not sure what that might be or what I should attempt to do next.
Any help would be great.
-
What do you get when you key in
ps aux|grep mysqlor ps aux|grep mariadb`? One of those commands should output to show if the sql server is running in memory.If its not running in memory then issue the
systemctl start mariadbIf it ends up in a failure state then look at the log files in /var/log to see why the database is having issues starting. -
@george1421 Hey George, see below for the outputs to your questions.
I’m not sure if the mariadb is separate to the mysql one or together? That’s what I get when I try to run the command though.
In terms of the log file, I’m seeing the below -
Cat seems to be the right command but doesn’t open anything when I run it? -
@dylan123 We the new version of mysql is called mariadb since Oracle has taken mysql closed source mariadb was forked from the last public version of mysql. So that is where the confusion is. All of the mysql commands work with mariadb.
So it almost looks like mysql is not installed on your fog server. When you do a
sytemctl start mysqldo you also get the service is not found?In regards to the log files you are only interested in the .log file the .gz files are older archived log files.
What host OS is your FOG server running?
-
@george1421 Hey George,
I assume the command should be systemctl?
When I changed it to that it is just blank until I quit out of it -
Running Ubuntu version 16.
-
@dylan123 Yeah that was a type-o, sorry. So starting mysql doesn’t respond with anything?
What about restart? And then starting it doesn’t put anything in the error.log ? If so something is not right with your system.
-
@george1421 So gave it a restart and tried running the command again but it failed.
Result when running systemctl status mysql.service
Result when running journalctl -xe
The denied entries seem interesting, not that I know why they’d be getting denied but I imagine that might be the issue.
I get no return at all though when I run cat /var/log/mysql/error.log
Everything had been running fine for a couple of years so I’m confused at to what’s happened. Must have been an update it’s done by itself, just find it strange that restoring hasn’t worked.
-
@dylan123 Ok, mysql is not starting anymore. And it seems to be cause by the security framework apparmor as we see in the logs output. I think I have seen this on one of my test machines once as well but I don’t know why it’s caused. Probably some part of package updates that is done when you ran the FOG installer.
Quite possible there was a different issue with mysql at first and re-running the FOG installer made it worse?!
See if you can make it work using this: https://support.plesk.com/hc/en-us/articles/360004185293-Unable-to-start-MySQL-on-Ubuntu-AVC-apparmor-DENIED-operation-open-
-
@Sebastian-Roth I gave it a crack but not luck. Well it seemed to get rid of the permission errors but I just got generic failed to start -
– The result is failed.
Oct 29 19:19:24 Fog sudo[1883]: pam_unix(sudo:session): session closed for user
Oct 29 19:19:24 Fog systemd[1]: mysql.service: Unit entered failed state.
Oct 29 19:19:24 Fog systemd[1]: mysql.service: Failed with result ‘exit-code’.
Oct 29 19:19:24 Fog systemd[1]: mysql.service: Service hold-off time over, sched
Oct 29 19:19:24 Fog systemd[1]: Stopped MySQL Community Server.
– Subject: Unit mysql.service has finished shutting downI’m trying to restore from a VM took 37 days ago, hopefully that does the trick… Will report back.
-
@dylan123 said in Database Connection Unavailable:
I’m trying to restore from a VM took 37 days ago, hopefully that does the trick… Will report back.
Just confirming that this backup worked thankfully so that’s a good result.
Thanks for the time and efforts as always guys, much appreciated!
-
@dylan123 I was just thinking about it on the way into the office this AM. Being out of space on the root partition would keep mysql from starting. I’ve done it my self. Now that you restored it from backup that evidence is kind of gone. But if you issue a
df -hlets see what your free space is now? -
@dylan123 What about the error logs??
If I remember correctly I did reset my VM to an earlier state as well just to get rid of the issue. But I’d really be interested to know how this happens. I find it really strange.
-
@george1421 said in Database Connection Unavailable:
@dylan123 I was just thinking about it on the way into the office this AM. Being out of space on the root partition would keep mysql from starting. I’ve done it my self. Now that you restored it from backup that evidence is kind of gone. But if you issue a
df -hlets see what your free space is now?Here’s the result of that -
I’m not sure which of the outputs the database sits on to know whether it’s out of space or not.
I still have a backup of where it doesn’t seem to work as my first restore had the same issue so I can bring that back temporarily to do some testing/troubleshooting to try pin it down further.
@Sebastian-Roth said in Database Connection Unavailable:
@dylan123 What about the error logs??
If I remember correctly I did reset my VM to an earlier state as well just to get rid of the issue. But I’d really be interested to know how this happens. I find it really strange.
If I run cat /var/log/mysql/log.error I still get a no result return. Running journalctl -xe doesn’t show up any of the errors it previously was, it ends with ‘the start-up result is done.’
-
@dylan123 Sometimes log rotation moves away current log infos. So when you get to take a look at the corrupted state again, see if you can gather some log information by running
zcat /var/log/mysql/error.log.1.gz(...2.gzand so on) -
@dylan123 Well looking at the output it looks like everything is mounted on your root partition. You have ~90GB free space on your entire server. Its possible that 2 system uploads could have filled up your root partition since your fog images are on the same partition as the OS. Roughly equivalent to storing your images on the
drive and filling that up. -
@dylan123
typically mariadb defaults to/var/lib/mysql/for the data files.It looks like your boot partition may be full. While this wouldn’t necessarily cause you issues with day to day operation, it can play havoc with apt if you have a kernel update in the queue when running updates. I have had more than several update sequences bork due to this, and depending on where it borked, I could see packages that have a number of dependencies such as mariadb being left in a non-startable state. I would suggest doing an
apt autoremoveto remove the unneeded kernels, but be aware you may need to manually give apt a little room in /boot by manually deleting an old kernel or two then running aapt -f install(I think) to cleanup from the dirty state. I believe Ubuntu generally keeps the current and one or two previous kernels by default.If you want to see if this might be an issue prior to digging around, a
apt upgradeshould complain at you. -
@dylan123 Reading a bit more about the apparmor stuff I found this: https://stackoverflow.com/a/49583958
I read in another SO thread comment that the apparmor=“DENIED” message probably isn’t the reason that MySQL (or in my case MariaDB) wasn’t starting, as it’s only a warning.
Confirmed by this bug report:
On the affected system, there was no noticeable impact (yet?) other than the denials, so I’d say it’s low impact.
Spinning up a VM with Ubuntu 18.04 I saw that quite true it has those log messages in
/var/log/syslogon my VM as well but the DB is running perfectly fine. So just for the records: Those apparmor=“DENIED” message don’t cause an impact on the DB and can be ignored.Check all the information George and Daniel posted as well as my suggestion on the mysql logs.
