CentOS 7 Drive Full

[Marcus Allen]

Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   30G   30G  303M 100% /
devtmpfs                 911M     0  911M   0% /dev
tmpfs                    921M     0  921M   0% /dev/shm
tmpfs                    921M  8.5M  912M   1% /run
tmpfs                    921M     0  921M   0% /sys/fs/cgroup
/dev/sda1                497M  120M  378M  24% /boot

[Wayne Workman]

@Marcus-Allen

Is that correct? you basically only have 30 GB available on this server?

[Marcus Allen]

Thats correct. Its a VM, I can increase the disk space if needed, however last week I was using about 11GB, so I didnt bump it up yet.

[Marcus Allen]

@Wayne-Workman

I’m bumping up to 64gb. This is a FOG Only server, no images are stored here.

[Wayne Workman]

@Marcus-Allen I just tried this, it worked better than the other one:

find . -type f -printf "%s\t%p\n" | sort -n | tail -10

[Marcus Allen]

@Wayne-Workman

15773912        ./var/www/html/fog/service/ipxe/init.xz
15775828        ./var/www/html/fog.prev/service/ipxe/init.xz
18102074        ./boot/initramfs-3.10.0-229.el7.x86_64.img
18179668        ./boot/initramfs-3.10.0-229.el7.x86_64kdump.img
18874368        ./var/lib/mysql/ibdata1
24281088        ./var/lib/rpm/Packages
39835970        ./boot/initramfs-0-rescue-e248d8e9ddcb47a99cdece8ee62fabc5.img
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc
140737486266368 ./proc/kcore

[Wayne Workman]

@Marcus-Allen I think it’s obvious where your space went…

[Marcus Allen]

@Marcus-Allen said:

proc/kcore

Got it

[Wayne Workman]

@Marcus-Allen Just following up on this thread, @Wolfbane8653 was correct (ofcourse).

But, on Red-Hat based distros, the path to the logs is in

/var/log/httpd

I just had to deal with this issue at one of my sites… fog server had a 300GB Apache log file lol. I deleted the file and then restarted httpd. Some of the tables in the fog DB were marked as crashed as well. I had to issue a repair statement on those to fix them like this:

mysqlcheck --repair --databases fog

and then I gave mysql a restart and all was fine.

[Marcus Allen]

@Wayne-Workman said:

Sorry, I’m a touch confused now. My logs are empty in the /var/log/httpd directory. I even cleaned them out too just to be safe, still only about 10MB.

Or, were you just mentioning about the log files in general?

[Wayne Workman]

@Marcus-Allen There is a fog bug that causes the Apache log to grow out of control in some revisions. That’s what we’re talking about.

The /proc/kcore that you found earlier was also on the machine I fixed earlier. But then I noticed the 300 GB log file in /var/log/httpd

[Wayne Workman]

@Marcus-Allen said:

@Wayne-Workman

15773912        ./var/www/html/fog/service/ipxe/init.xz
15775828        ./var/www/html/fog.prev/service/ipxe/init.xz
18102074        ./boot/initramfs-3.10.0-229.el7.x86_64.img
18179668        ./boot/initramfs-3.10.0-229.el7.x86_64kdump.img
18874368        ./var/lib/mysql/ibdata1
24281088        ./var/lib/rpm/Packages
39835970        ./boot/initramfs-0-rescue-e248d8e9ddcb47a99cdece8ee62fabc5.img
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc
140737486266368 ./proc/kcore

Have you examined those two files? They are quite large.

67108864 ./sys/devices/pci0000:00/0000:00:0f.0/resource1
67108864 ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc

Also, if you issue this command from / (from root) it might find more.

cd /
find . -type f -printf "%s\t%p\n" | sort -n | tail -10

[Wayne Workman]

@Marcus-Allen Look inside of /var/log I had some pretty big files in there too.