CentOS 7 Drive Full



  • So, I came into work today and the fog service was stopped. Rebooted the machine and saw the message that it couldnt write to the disk.

    Result of df

    Filesystem              1K-blocks     Used Available Use% Mounted on
    /dev/mapper/centos-root  31340544 31015780    324764  99% /
    devtmpfs                   932704        0    932704   0% /dev
    tmpfs                      942348        0    942348   0% /dev/shm
    tmpfs                      942348     8608    933740   1% /run
    tmpfs                      942348        0    942348   0% /sys/fs/cgroup
    /dev/sda1                  508588   122016    386572  24% /boot
    
    

    Result of du -sh

    du: cannot access ‘./proc/12278/task/12278/fd/4’: No such file or directory
    du: cannot access ‘./proc/12278/task/12278/fdinfo/4’: No such file or directory
    du: cannot access ‘./proc/12278/fd/4’: No such file or directory
    du: cannot access ‘./proc/12278/fdinfo/4’: No such file or directory
    1.6G    .
    

    I cannot figure out where these “files” are.



  • @Marcus-Allen Look inside of /var/log I had some pretty big files in there too.



  • @Marcus-Allen said:

    @Wayne-Workman

    15773912        ./var/www/html/fog/service/ipxe/init.xz
    15775828        ./var/www/html/fog.prev/service/ipxe/init.xz
    18102074        ./boot/initramfs-3.10.0-229.el7.x86_64.img
    18179668        ./boot/initramfs-3.10.0-229.el7.x86_64kdump.img
    18874368        ./var/lib/mysql/ibdata1
    24281088        ./var/lib/rpm/Packages
    39835970        ./boot/initramfs-0-rescue-e248d8e9ddcb47a99cdece8ee62fabc5.img
    67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1
    67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc
    140737486266368 ./proc/kcore
    
    

    Have you examined those two files? They are quite large.

    67108864 ./sys/devices/pci0000:00/0000:00:0f.0/resource1
    67108864 ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc

    Also, if you issue this command from / (from root) it might find more.

    cd /
    find . -type f -printf "%s\t%p\n" | sort -n | tail -10
    


  • @Marcus-Allen There is a fog bug that causes the Apache log to grow out of control in some revisions. That’s what we’re talking about.

    The /proc/kcore that you found earlier was also on the machine I fixed earlier. But then I noticed the 300 GB log file in /var/log/httpd



  • @Wayne-Workman said:

    Sorry, I’m a touch confused now. My logs are empty in the /var/log/httpd directory. I even cleaned them out too just to be safe, still only about 10MB.

    Or, were you just mentioning about the log files in general?



  • @Marcus-Allen Just following up on this thread, @Wolfbane8653 was correct (ofcourse).

    But, on Red-Hat based distros, the path to the logs is in

    /var/log/httpd

    I just had to deal with this issue at one of my sites… fog server had a 300GB Apache log file lol. I deleted the file and then restarted httpd. Some of the tables in the fog DB were marked as crashed as well. I had to issue a repair statement on those to fix them like this:

    mysqlcheck --repair --databases fog
    

    and then I gave mysql a restart and all was fine.



  • @Marcus-Allen said:

    proc/kcore

    Got it



  • @Marcus-Allen I think it’s obvious where your space went…



  • @Wayne-Workman

    15773912        ./var/www/html/fog/service/ipxe/init.xz
    15775828        ./var/www/html/fog.prev/service/ipxe/init.xz
    18102074        ./boot/initramfs-3.10.0-229.el7.x86_64.img
    18179668        ./boot/initramfs-3.10.0-229.el7.x86_64kdump.img
    18874368        ./var/lib/mysql/ibdata1
    24281088        ./var/lib/rpm/Packages
    39835970        ./boot/initramfs-0-rescue-e248d8e9ddcb47a99cdece8ee62fabc5.img
    67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1
    67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc
    140737486266368 ./proc/kcore
    
    


  • @Marcus-Allen I just tried this, it worked better than the other one:

    find . -type f -printf "%s\t%p\n" | sort -n | tail -10
    


  • @Wayne-Workman

    I’m bumping up to 64gb. This is a FOG Only server, no images are stored here.



  • Thats correct. Its a VM, I can increase the disk space if needed, however last week I was using about 11GB, so I didnt bump it up yet.



  • @Marcus-Allen

    Is that correct? you basically only have 30 GB available on this server?



  • Filesystem               Size  Used Avail Use% Mounted on
    /dev/mapper/centos-root   30G   30G  303M 100% /
    devtmpfs                 911M     0  911M   0% /dev
    tmpfs                    921M     0  921M   0% /dev/shm
    tmpfs                    921M  8.5M  912M   1% /run
    tmpfs                    921M     0  921M   0% /sys/fs/cgroup
    /dev/sda1                497M  120M  378M  24% /boot
    
    


  • @Marcus-Allen Ok. Your last post you put up was in human readable format so do it that way.

    Can you post this?

    df -h
    


  • @Wayne-Workman said:

    du -a -h / | sort -n -r -h | head -n 10

    sort: options '-hn' are incompatible
    


  •  du -a -h / | sort -n -r -h | head -n 10
    

    or this:

     du -a -h / | sort -nrh | head -n 10 
    


  • @Wayne-Workman

    I dont think that is working right:

    du -a -h / | sort -n -r | head -n 10
    1020K   /usr/lib/kbd/consolefonts
    1016K   /usr/share/doc/sudo-1.8.6p7
    1016K   /usr/lib64/python2.7/site-packages/gi
    1012K   /usr/share/pki/ca-trust-source
    1012K   /usr/share/pki
    1012K   /usr/lib/udev/hwdb.d/20-usb-vendor-model.hwdb
    1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko
    1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x
    1008K   /usr/lib/firmware/cxgb4
    1004K   /var/lib/mysql/mysql
    

    Is that re right spot to put that switch in?



  • @Marcus-Allen said:

    @Wayne-Workman

    Here is the output:

    1621736 /
    975920  /usr
    404516  /var
    296068  /usr/share
    268040  /var/lib
    225724  /usr/lib
    199072  /var/lib/php/session
    199072  /var/lib/php
    167288  /usr/lib64
    155356  /usr/bin
    

    Here is the /usr folder

    975920  /usr
    296068  /usr/share
    225724  /usr/lib
    167288  /usr/lib64
    155356  /usr/bin
    113044  /usr/lib/modules/3.10.0-229.el7.x86_64
    113044  /usr/lib/modules
    110424  /usr/lib/modules/3.10.0-229.el7.x86_64/kernel
    99992   /usr/share/locale
    78648   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers
    

    add -h to the sort command.

    -h gives you human readable sizes.



  • @Wayne-Workman

    Here is the output:

    1621736 /
    975920  /usr
    404516  /var
    296068  /usr/share
    268040  /var/lib
    225724  /usr/lib
    199072  /var/lib/php/session
    199072  /var/lib/php
    167288  /usr/lib64
    155356  /usr/bin
    

    Here is the /usr folder

    975920  /usr
    296068  /usr/share
    225724  /usr/lib
    167288  /usr/lib64
    155356  /usr/bin
    113044  /usr/lib/modules/3.10.0-229.el7.x86_64
    113044  /usr/lib/modules
    110424  /usr/lib/modules/3.10.0-229.el7.x86_64/kernel
    99992   /usr/share/locale
    78648   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers
    

 

521
Online

5.4k
Users

12.6k
Topics

118.7k
Posts