Unable to restart MariaDB

[alomarh]

Hi everyone.

I wanted to log in to the webUI of my FOG server to check on an image but it came back with a “Database connection unavailable” error on the browser. I logged in to the server and when I ran “systemctl restart mysql” it came back with this error:

Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.

so I ran “systemctl status mariadb.service” and this is what it’s coming back to me with:

Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
root@FOG-SERVER:~# systemctl status mariadb.service
● mariadb.service - MariaDB 10.3.27 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2021-08-18 11:04:19 MDT; 5s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 9243 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
  Process: 9244 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 9246 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=
  Process: 9290 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
 Main PID: 9290 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"

Aug 18 11:04:13 FOG-SERVER systemd[1]: Starting MariaDB 10.3.27 database server...
Aug 18 11:04:13 FOG-SERVER mysqld[9290]: 2021-08-18 11:04:13 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 9290 ...
Aug 18 11:04:19 FOG-SERVER systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 18 11:04:19 FOG-SERVER systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 18 11:04:19 FOG-SERVER systemd[1]: Failed to start MariaDB 10.3.27 database server.

I found a couple of “fixes” on Google searches that has not helped. Can anyone tell me what I’m missing here? Or maybe point to the right direction for troubleshooting this?

Re-installing the database is my last resort and I’m willing to if I have to but can anyone advise what the process for that would be? I have 6 laptops and 2 desktop that I need to image but with my FOG server being down at the moment, I am stuck in the water. Any help will be greatly appreciated. Thanks in advance.

[george1421]

@alomarh on the fog linux server console run this command df -h to see if you are out of disk space on your root volume. Post the results of that command here.

[alomarh]

@george1421 This is what I get. Seems like that would be the case.

Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           391M   40M  351M  11% /run
/dev/sda2      1006G  982G     0 100% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda1       511M  5.2M  506M   2% /boot/efi
tmpfs           391M     0  391M   0% /run/user/1000

[george1421]

@alomarh Yes your root partition is full so the db can’t start.

This is the risk when you capture images to the default location. (not something you selected, but how FOG is setup by default).

First start to look for space in /image/dev if there are any directories in there that have a mac looking like address you can purge those. They are left behind because of a botched image capture. Then go after image files in /images/<image_name> to see if you can discard some old images. Once you get the database up and running again then you can think about how to keep this from happening again.

[alomarh]

@george1421 Thank you George. That solved it. I know I deleted images before from the web interface but when I looked at /images I saw the directories were still there with the files intact. So I manually removed everything I needed removed. Is that a bug in the web interface?

[george1421]

@alomarh Well as they say in the funny papers, its not a bug but a feature. In this case its true, but its not documented very well.

If you delete the image from the list view of images, it only removed the database record for the images (to avoid accidental purging of the data files). If you go into the image definition and delete it there, you are prompted to remove the data files. I really think when you delete it from the list, the system should ask you if you want to remove the image files. Maybe that can make it into the 1.6.x branch of the code. The system is working as it was designed, just how one might expect.

[alomarh]

@george1421 Thank you for the clarification. This is duly noted. Appreciate the help, you saved my ass today basically.