MySQL Failed

rets34 · Jul 29, 2020, 4:14 PM

Ubuntu 18.04 | 4.15.0-72generic
Attempting to upgrade fog server to 1.5.9-RC2 worked fine the first time, but after a reboot I got the ‘database connection unavailable’ so I went to run installfog.sh to fix it, and was met with the following issue.
I went and ran installfog.sh and everything goes fine until the MySQL part of the installation.

* Setting up and starting MySQL...............................Failed!

I’ve tried purging MySQL and the MariDB but I keep getting the same issue.

fog_error_1.5.9-RC2.log

Sebastian Roth · Aug 1, 2020, 8:57 AM

@rets34 Size on client is only being updated when re-capturing the image. Don’t worry about it.

george1421 · Jul 29, 2020, 4:17 PM

Mod Note: Moving to the bug reports forum.

Tom Elliott · Jul 29, 2020, 9:17 PM

I’m having a hard time seeing this as a bug.

What’s the disk space available on the machine?

what’s the output of journalctl -xe?

What’s the output of df -h

rets34 · Jul 30, 2020, 3:36 PM

Boot drive has 295GB free

Output of df -h:

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           791M  1.7M  789M   1% /run
/dev/sda2       292G   18G  260G   7% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0      1.0M  1.0M     0 100% /snap/gnome-logs/100
/dev/loop3      2.3M  2.3M     0 100% /snap/gnome-system-monitor/148
/dev/loop1      2.5M  2.5M     0 100% /snap/gnome-calculator/730
/dev/loop14     2.5M  2.5M     0 100% /snap/gnome-calculator/748
/dev/loop5      384K  384K     0 100% /snap/gnome-characters/539
/dev/loop6      384K  384K     0 100% /snap/gnome-characters/550
/dev/loop7       94M   94M     0 100% /snap/core/9066
/dev/loop8      1.0M  1.0M     0 100% /snap/gnome-logs/93
/dev/loop10      55M   55M     0 100% /snap/core18/1754
/dev/loop9      162M  162M     0 100% /snap/gnome-3-28-1804/128
/dev/loop12      55M   55M     0 100% /snap/gtk-common-themes/1502
/dev/loop16     141M  141M     0 100% /snap/gnome-3-26-1604/98
/dev/loop19      63M   63M     0 100% /snap/gtk-common-themes/1506
/dev/loop18     2.3M  2.3M     0 100% /snap/gnome-system-monitor/145
/dev/loop13     256M  256M     0 100% /snap/gnome-3-34-1804/33
/dev/loop11      97M   97M     0 100% /snap/core/9665
/dev/loop2      256M  256M     0 100% /snap/gnome-3-34-1804/36
/dev/loop17     161M  161M     0 100% /snap/gnome-3-28-1804/116
/dev/loop4      141M  141M     0 100% /snap/gnome-3-26-1604/100
/dev/loop15      55M   55M     0 100% /snap/core18/1880
/dev/sdb1       916G  447G  424G  52% /media/admin/Storage
/dev/sdc1       916G  357G  513G  42% /images
/dev/sda1       511M  2.7M  509M   1% /boot/efi
tmpfs           791M   56K  791M   1% /run/user/1001

I will attach the output of journalctl -xe, as it is a bit of a long mess.
journalctl.txt

Tom Elliott · Jul 30, 2020, 4:54 PM

@rets34 Please try:

systemctl restart mariadb
journalctl -xe
systemctl -l status mariadb

rets34 · Jul 30, 2020, 5:20 PM

@Tom-Elliott

Attached output below.

mariadb.txt

Tom Elliott · Jul 30, 2020, 5:23 PM

@rets34 I’m not seeing anything that stands out as problemattic.

It appears mariadb started okay, though I’m unsure.

Can you also provide systemctl -l status mariadb

rets34 · Jul 30, 2020, 6:08 PM

@Tom-Elliott

root@fog-desktop:/home/admin# systemctl -l status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: activating (start) since Thu 2020-07-30 12:16:53 CDT; 34s ago
  Process: 30832 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_rec
  Process: 30829 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=
  Process: 30828 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0
 Main PID: 30916 (mysqld)
    Tasks: 26 (limit: 4915)
   CGroup: /system.slice/mariadb.service
           └─30916 /usr/sbin/mysqld

Jul 30 12:16:53 fog-desktop systemd[1]: Starting MariaDB database server...
Jul 30 12:16:53 fog-desktop mysqld[30916]: 2020-07-30 12:16:53 139648654302336 [Note] /usr/sbin/mysqld (mysqld 1
root@fog-desktop:/home/admin# ^C
root@fog-desktop:/home/admin# systemctl -l status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Thu 2020-07-30 12:18:26 CDT; 50min ago
  Process: 30916 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited
  Process: 30832 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_rec
  Process: 30829 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=
  Process: 30828 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0
 Main PID: 30916 (code=exited, status=0/SUCCESS)

Jul 30 12:16:53 fog-desktop systemd[1]: Starting MariaDB database server...
Jul 30 12:16:53 fog-desktop mysqld[30916]: 2020-07-30 12:16:53 139648654302336 [Note] /usr/sbin/mysqld (mysqld 1
Jul 30 12:18:23 fog-desktop systemd[1]: mariadb.service: Start operation timed out. Terminating.
Jul 30 12:18:26 fog-desktop systemd[1]: mariadb.service: Failed with result 'timeout'.
Jul 30 12:18:26 fog-desktop systemd[1]: Failed to start MariaDB database server.
root@fog-desktop:/home/admin#

Tom Elliott · Jul 30, 2020, 6:29 PM

@rets34 said in MySQL Failed:

systemd[1]: mariadb.service: Failed with result ‘timeout’.

Please follow this information:
https://bugs.launchpad.net/ubuntu/+source/mariadb-10.1/+bug/1806263

The gist:

echo "/usr/sbin/mysqld { }" > /etc/apparmor.d/usr.sbin.mysqld
apparmor_parser -v -R /etc/apparmor.d/usr.sbin.mysqld
systemctl restart mariadb

Then wait a couple of seconds and do the systemctl -l status mariadb

rets34 · Jul 30, 2020, 7:58 PM

root@fog-desktop:/home/admin# systemctl -l status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-07-30 13:37:37 CDT; 17s ago
  Process: 23218 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status
  Process: 23215 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
  Process: 23104 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_rec
  Process: 23102 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=
  Process: 23101 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0
 Main PID: 23188 (mysqld)
   Status: "Taking your SQL requests now..."
    Tasks: 29 (limit: 4915)
   CGroup: /system.slice/mariadb.service
           └─23188 /usr/sbin/mysqld

Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: Processing databases
Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: information_schema
Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: mysql
Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: performance_schema
Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: Phase 6/7: Checking and upgrading tables
Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: Processing databases
Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: information_schema
Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: performance_schema
Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: Phase 7/7: Running 'FLUSH PRIVILEGES'
Jul 30 13:37:39 fog-desktop /etc/mysql/debian-start[23221]: OK

That fixed it, however after a reboot I got ‘database connection unavailable’ again.
And back to square one with:

 Setting up and starting MySQL...............................Failed!

Tom Elliott · Jul 31, 2020, 1:20 AM

@rets34 you probably need to remove MySQL proper. apt-get remove --purge mysql-server

Followed by the commands I gave below.

I gave the gist to get you up and running right away. Why you would reboot again, when that broke this the first time, seems strange. This is a server and should not need to be restarted every time you want to update fog.

rets34 · Jul 31, 2020, 1:53 PM

@Tom-Elliott I was just if I had to reboot the machine for whatever reason that I wouldn’t have to worry about it failing. But the solution you posted has worked to get it working.
Now I just need to figure out how to get all my images to show up in the WebUI, got them all on the machine but don’t have an exported CSV from prior to database dying.

Sebastian Roth · Jul 31, 2020, 5:37 PM

@rets34 said in MySQL Failed:

Now I just need to figure out how to get all my images to show up in the WebUI, got them all on the machine but don’t have an exported CSV from prior to database dying.

You can simply re-create the image definitions in the web UI and make sure they are pointed to the existing directories. They won’t be overwritten unless you re-capture the image! It’s always good to keep a full daily database backup or at least pull a CSV dump every now and then.

rets34 · Jul 31, 2020, 6:54 PM

@Sebastian-Roth

I went through and pointed a couple to the old location, and it’s not showing Size?

Sebastian Roth · Aug 1, 2020, 8:57 AM

@rets34 Size on client is only being updated when re-capturing the image. Don’t worry about it.

rets34 · Aug 4, 2020, 3:59 PM

@Sebastian-Roth Makes sense to me.
Solved for me

MySQL Failed

156

12.0k

17.3k

155.2k