CentOS 7 Drive Full

[Marcus Allen]

@Wayne-Workman

Here is the output:

1621736 /
975920  /usr
404516  /var
296068  /usr/share
268040  /var/lib
225724  /usr/lib
199072  /var/lib/php/session
199072  /var/lib/php
167288  /usr/lib64
155356  /usr/bin

Here is the /usr folder

975920  /usr
296068  /usr/share
225724  /usr/lib
167288  /usr/lib64
155356  /usr/bin
113044  /usr/lib/modules/3.10.0-229.el7.x86_64
113044  /usr/lib/modules
110424  /usr/lib/modules/3.10.0-229.el7.x86_64/kernel
99992   /usr/share/locale
78648   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers

[Wayne Workman]

@Marcus-Allen said:

@Wayne-Workman

Here is the output:

1621736 /
975920  /usr
404516  /var
296068  /usr/share
268040  /var/lib
225724  /usr/lib
199072  /var/lib/php/session
199072  /var/lib/php
167288  /usr/lib64
155356  /usr/bin

Here is the /usr folder

975920  /usr
296068  /usr/share
225724  /usr/lib
167288  /usr/lib64
155356  /usr/bin
113044  /usr/lib/modules/3.10.0-229.el7.x86_64
113044  /usr/lib/modules
110424  /usr/lib/modules/3.10.0-229.el7.x86_64/kernel
99992   /usr/share/locale
78648   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers

add -h to the sort command.

-h gives you human readable sizes.

[Marcus Allen]

@Wayne-Workman

I dont think that is working right:

du -a -h / | sort -n -r | head -n 10
1020K   /usr/lib/kbd/consolefonts
1016K   /usr/share/doc/sudo-1.8.6p7
1016K   /usr/lib64/python2.7/site-packages/gi
1012K   /usr/share/pki/ca-trust-source
1012K   /usr/share/pki
1012K   /usr/lib/udev/hwdb.d/20-usb-vendor-model.hwdb
1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko
1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x
1008K   /usr/lib/firmware/cxgb4
1004K   /var/lib/mysql/mysql

Is that re right spot to put that switch in?

[Wayne Workman]

 du -a -h / | sort -n -r -h | head -n 10

or this:

 du -a -h / | sort -nrh | head -n 10 

[Marcus Allen]

@Wayne-Workman said:

du -a -h / | sort -n -r -h | head -n 10

sort: options '-hn' are incompatible

[Wayne Workman]

@Marcus-Allen Ok. Your last post you put up was in human readable format so do it that way.

Can you post this?

df -h

[Marcus Allen]

Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   30G   30G  303M 100% /
devtmpfs                 911M     0  911M   0% /dev
tmpfs                    921M     0  921M   0% /dev/shm
tmpfs                    921M  8.5M  912M   1% /run
tmpfs                    921M     0  921M   0% /sys/fs/cgroup
/dev/sda1                497M  120M  378M  24% /boot

[Wayne Workman]

@Marcus-Allen

Is that correct? you basically only have 30 GB available on this server?

[Marcus Allen]

Thats correct. Its a VM, I can increase the disk space if needed, however last week I was using about 11GB, so I didnt bump it up yet.

[Marcus Allen]

@Wayne-Workman

I’m bumping up to 64gb. This is a FOG Only server, no images are stored here.

[Wayne Workman]

@Marcus-Allen I just tried this, it worked better than the other one:

find . -type f -printf "%s\t%p\n" | sort -n | tail -10

[Marcus Allen]

@Wayne-Workman

15773912        ./var/www/html/fog/service/ipxe/init.xz
15775828        ./var/www/html/fog.prev/service/ipxe/init.xz
18102074        ./boot/initramfs-3.10.0-229.el7.x86_64.img
18179668        ./boot/initramfs-3.10.0-229.el7.x86_64kdump.img
18874368        ./var/lib/mysql/ibdata1
24281088        ./var/lib/rpm/Packages
39835970        ./boot/initramfs-0-rescue-e248d8e9ddcb47a99cdece8ee62fabc5.img
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc
140737486266368 ./proc/kcore

[Wayne Workman]

@Marcus-Allen I think it’s obvious where your space went…

[Marcus Allen]

@Marcus-Allen said:

proc/kcore

Got it

[Wayne Workman]

@Marcus-Allen Just following up on this thread, @Wolfbane8653 was correct (ofcourse).

But, on Red-Hat based distros, the path to the logs is in

/var/log/httpd

I just had to deal with this issue at one of my sites… fog server had a 300GB Apache log file lol. I deleted the file and then restarted httpd. Some of the tables in the fog DB were marked as crashed as well. I had to issue a repair statement on those to fix them like this:

mysqlcheck --repair --databases fog

and then I gave mysql a restart and all was fine.

[Marcus Allen]

@Wayne-Workman said:

Sorry, I’m a touch confused now. My logs are empty in the /var/log/httpd directory. I even cleaned them out too just to be safe, still only about 10MB.

Or, were you just mentioning about the log files in general?

[Wayne Workman]

@Marcus-Allen There is a fog bug that causes the Apache log to grow out of control in some revisions. That’s what we’re talking about.

The /proc/kcore that you found earlier was also on the machine I fixed earlier. But then I noticed the 300 GB log file in /var/log/httpd

[Wayne Workman]

@Marcus-Allen said:

@Wayne-Workman

15773912        ./var/www/html/fog/service/ipxe/init.xz
15775828        ./var/www/html/fog.prev/service/ipxe/init.xz
18102074        ./boot/initramfs-3.10.0-229.el7.x86_64.img
18179668        ./boot/initramfs-3.10.0-229.el7.x86_64kdump.img
18874368        ./var/lib/mysql/ibdata1
24281088        ./var/lib/rpm/Packages
39835970        ./boot/initramfs-0-rescue-e248d8e9ddcb47a99cdece8ee62fabc5.img
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc
140737486266368 ./proc/kcore

Have you examined those two files? They are quite large.

67108864 ./sys/devices/pci0000:00/0000:00:0f.0/resource1
67108864 ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc

Also, if you issue this command from / (from root) it might find more.

cd /
find . -type f -printf "%s\t%p\n" | sort -n | tail -10

[Wayne Workman]

@Marcus-Allen Look inside of /var/log I had some pretty big files in there too.