CentOS 7 Drive Full

[Marcus Allen]

So, I came into work today and the fog service was stopped. Rebooted the machine and saw the message that it couldnt write to the disk.

Result of df

Filesystem              1K-blocks     Used Available Use% Mounted on
/dev/mapper/centos-root  31340544 31015780    324764  99% /
devtmpfs                   932704        0    932704   0% /dev
tmpfs                      942348        0    942348   0% /dev/shm
tmpfs                      942348     8608    933740   1% /run
tmpfs                      942348        0    942348   0% /sys/fs/cgroup
/dev/sda1                  508588   122016    386572  24% /boot

Result of du -sh

du: cannot access ‘./proc/12278/task/12278/fd/4’: No such file or directory
du: cannot access ‘./proc/12278/task/12278/fdinfo/4’: No such file or directory
du: cannot access ‘./proc/12278/fd/4’: No such file or directory
du: cannot access ‘./proc/12278/fdinfo/4’: No such file or directory
1.6G    .

I cannot figure out where these “files” are.

[Wolfbane8653]

Apache logs consuming fog server hdd (maxes to 100%)
–Have to delete everything in /var/log/apache
–then restart service to clear hdd on server
–finally HDD is back to 22% used

This is a current issue with the SVN

cd /var/log/apache/
rm *
service apache2 restart

[Marcus Allen]

Unfortunately I have tried that and the apache logs are VERY small right now.

total 16K
-rw-r--r-- 1 root root 8.8K Jul 20 14:31 access_log
-rw-r--r-- 1 root root  997 Jul 20 14:30 error_log
-rw-r--r-- 1 root root    0 Jul 20 14:30 ssl_access_log
-rw-r--r-- 1 root root    0 Jul 20 14:30 ssl_error_log
-rw-r--r-- 1 root root    0 Jul 20 14:30 ssl_request_log

I did go ahead and delete them, then reboot and still no luck.

[Wayne Workman]

@Marcus-Allen Check this out: http://www.cyberciti.biz/faq/how-do-i-find-the-largest-filesdirectories-on-a-linuxunixbsd-filesystem/

[Marcus Allen]

@Wayne-Workman

Here is the output:

1621736 /
975920  /usr
404516  /var
296068  /usr/share
268040  /var/lib
225724  /usr/lib
199072  /var/lib/php/session
199072  /var/lib/php
167288  /usr/lib64
155356  /usr/bin

Here is the /usr folder

975920  /usr
296068  /usr/share
225724  /usr/lib
167288  /usr/lib64
155356  /usr/bin
113044  /usr/lib/modules/3.10.0-229.el7.x86_64
113044  /usr/lib/modules
110424  /usr/lib/modules/3.10.0-229.el7.x86_64/kernel
99992   /usr/share/locale
78648   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers

[Wayne Workman]

@Marcus-Allen said:

@Wayne-Workman

Here is the output:

1621736 /
975920  /usr
404516  /var
296068  /usr/share
268040  /var/lib
225724  /usr/lib
199072  /var/lib/php/session
199072  /var/lib/php
167288  /usr/lib64
155356  /usr/bin

Here is the /usr folder

975920  /usr
296068  /usr/share
225724  /usr/lib
167288  /usr/lib64
155356  /usr/bin
113044  /usr/lib/modules/3.10.0-229.el7.x86_64
113044  /usr/lib/modules
110424  /usr/lib/modules/3.10.0-229.el7.x86_64/kernel
99992   /usr/share/locale
78648   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers

add -h to the sort command.

-h gives you human readable sizes.

[Marcus Allen]

@Wayne-Workman

I dont think that is working right:

du -a -h / | sort -n -r | head -n 10
1020K   /usr/lib/kbd/consolefonts
1016K   /usr/share/doc/sudo-1.8.6p7
1016K   /usr/lib64/python2.7/site-packages/gi
1012K   /usr/share/pki/ca-trust-source
1012K   /usr/share/pki
1012K   /usr/lib/udev/hwdb.d/20-usb-vendor-model.hwdb
1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko
1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x
1008K   /usr/lib/firmware/cxgb4
1004K   /var/lib/mysql/mysql

Is that re right spot to put that switch in?

[Wayne Workman]

 du -a -h / | sort -n -r -h | head -n 10

or this:

 du -a -h / | sort -nrh | head -n 10 

[Marcus Allen]

@Wayne-Workman said:

du -a -h / | sort -n -r -h | head -n 10

sort: options '-hn' are incompatible

[Wayne Workman]

@Marcus-Allen Ok. Your last post you put up was in human readable format so do it that way.

Can you post this?

df -h

[Marcus Allen]

Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   30G   30G  303M 100% /
devtmpfs                 911M     0  911M   0% /dev
tmpfs                    921M     0  921M   0% /dev/shm
tmpfs                    921M  8.5M  912M   1% /run
tmpfs                    921M     0  921M   0% /sys/fs/cgroup
/dev/sda1                497M  120M  378M  24% /boot

[Wayne Workman]

@Marcus-Allen

Is that correct? you basically only have 30 GB available on this server?

[Marcus Allen]

Thats correct. Its a VM, I can increase the disk space if needed, however last week I was using about 11GB, so I didnt bump it up yet.

[Marcus Allen]

@Wayne-Workman

I’m bumping up to 64gb. This is a FOG Only server, no images are stored here.

[Wayne Workman]

@Marcus-Allen I just tried this, it worked better than the other one:

find . -type f -printf "%s\t%p\n" | sort -n | tail -10

[Marcus Allen]

@Wayne-Workman

15773912        ./var/www/html/fog/service/ipxe/init.xz
15775828        ./var/www/html/fog.prev/service/ipxe/init.xz
18102074        ./boot/initramfs-3.10.0-229.el7.x86_64.img
18179668        ./boot/initramfs-3.10.0-229.el7.x86_64kdump.img
18874368        ./var/lib/mysql/ibdata1
24281088        ./var/lib/rpm/Packages
39835970        ./boot/initramfs-0-rescue-e248d8e9ddcb47a99cdece8ee62fabc5.img
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc
140737486266368 ./proc/kcore

[Wayne Workman]

@Marcus-Allen I think it’s obvious where your space went…

[Marcus Allen]

@Marcus-Allen said:

proc/kcore

Got it

[Wayne Workman]

@Marcus-Allen Just following up on this thread, @Wolfbane8653 was correct (ofcourse).

But, on Red-Hat based distros, the path to the logs is in

/var/log/httpd

I just had to deal with this issue at one of my sites… fog server had a 300GB Apache log file lol. I deleted the file and then restarted httpd. Some of the tables in the fog DB were marked as crashed as well. I had to issue a repair statement on those to fix them like this:

mysqlcheck --repair --databases fog

and then I gave mysql a restart and all was fine.

[Marcus Allen]

@Wayne-Workman said:

Sorry, I’m a touch confused now. My logs are empty in the /var/log/httpd directory. I even cleaned them out too just to be safe, still only about 10MB.

Or, were you just mentioning about the log files in general?