• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

CentOS 7 Drive Full

Scheduled Pinned Locked Moved Solved
Linux Problems
3
23
11.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    Marcus Allen
    last edited by Marcus Allen Jul 20, 2015, 12:30 PM Jul 20, 2015, 6:23 PM

    So, I came into work today and the fog service was stopped. Rebooted the machine and saw the message that it couldnt write to the disk.

    Result of df

    Filesystem              1K-blocks     Used Available Use% Mounted on
    /dev/mapper/centos-root  31340544 31015780    324764  99% /
    devtmpfs                   932704        0    932704   0% /dev
    tmpfs                      942348        0    942348   0% /dev/shm
    tmpfs                      942348     8608    933740   1% /run
    tmpfs                      942348        0    942348   0% /sys/fs/cgroup
    /dev/sda1                  508588   122016    386572  24% /boot
    
    

    Result of du -sh

    du: cannot access ‘./proc/12278/task/12278/fd/4’: No such file or directory
    du: cannot access ‘./proc/12278/task/12278/fdinfo/4’: No such file or directory
    du: cannot access ‘./proc/12278/fd/4’: No such file or directory
    du: cannot access ‘./proc/12278/fdinfo/4’: No such file or directory
    1.6G    .
    

    I cannot figure out where these “files” are.

    W 1 Reply Last reply Jul 20, 2015, 6:41 PM Reply Quote 0
    • W
      Wolfbane8653 Developer
      last edited by Jul 20, 2015, 6:29 PM

      Apache logs consuming fog server hdd (maxes to 100%)
      –Have to delete everything in /var/log/apache
      –then restart service to clear hdd on server
      –finally HDD is back to 22% used

      This is a current issue with the SVN

      cd /var/log/apache/
      rm *
      service apache2 restart
      
      1 Reply Last reply Reply Quote 1
      • M
        Marcus Allen
        last edited by Jul 20, 2015, 6:39 PM

        Unfortunately I have tried that and the apache logs are VERY small right now.

        total 16K
        -rw-r--r-- 1 root root 8.8K Jul 20 14:31 access_log
        -rw-r--r-- 1 root root  997 Jul 20 14:30 error_log
        -rw-r--r-- 1 root root    0 Jul 20 14:30 ssl_access_log
        -rw-r--r-- 1 root root    0 Jul 20 14:30 ssl_error_log
        -rw-r--r-- 1 root root    0 Jul 20 14:30 ssl_request_log
        

        I did go ahead and delete them, then reboot and still no luck.

        1 Reply Last reply Reply Quote 0
        • W
          Wayne Workman @Marcus Allen
          last edited by Jul 20, 2015, 6:41 PM

          @Marcus-Allen Check this out: http://www.cyberciti.biz/faq/how-do-i-find-the-largest-filesdirectories-on-a-linuxunixbsd-filesystem/

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
          Daily Clean Installation Results:
          https://fogtesting.fogproject.us/
          FOG Reporting:
          https://fog-external-reporting-results.fogproject.us/

          M 1 Reply Last reply Jul 20, 2015, 6:43 PM Reply Quote 0
          • M
            Marcus Allen @Wayne Workman
            last edited by Jul 20, 2015, 6:43 PM

            @Wayne-Workman

            Here is the output:

            1621736 /
            975920  /usr
            404516  /var
            296068  /usr/share
            268040  /var/lib
            225724  /usr/lib
            199072  /var/lib/php/session
            199072  /var/lib/php
            167288  /usr/lib64
            155356  /usr/bin
            

            Here is the /usr folder

            975920  /usr
            296068  /usr/share
            225724  /usr/lib
            167288  /usr/lib64
            155356  /usr/bin
            113044  /usr/lib/modules/3.10.0-229.el7.x86_64
            113044  /usr/lib/modules
            110424  /usr/lib/modules/3.10.0-229.el7.x86_64/kernel
            99992   /usr/share/locale
            78648   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers
            
            W 1 Reply Last reply Jul 20, 2015, 6:45 PM Reply Quote 0
            • W
              Wayne Workman @Marcus Allen
              last edited by Jul 20, 2015, 6:45 PM

              @Marcus-Allen said:

              @Wayne-Workman

              Here is the output:

              1621736 /
              975920  /usr
              404516  /var
              296068  /usr/share
              268040  /var/lib
              225724  /usr/lib
              199072  /var/lib/php/session
              199072  /var/lib/php
              167288  /usr/lib64
              155356  /usr/bin
              

              Here is the /usr folder

              975920  /usr
              296068  /usr/share
              225724  /usr/lib
              167288  /usr/lib64
              155356  /usr/bin
              113044  /usr/lib/modules/3.10.0-229.el7.x86_64
              113044  /usr/lib/modules
              110424  /usr/lib/modules/3.10.0-229.el7.x86_64/kernel
              99992   /usr/share/locale
              78648   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers
              

              add -h to the sort command.

              -h gives you human readable sizes.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
              Daily Clean Installation Results:
              https://fogtesting.fogproject.us/
              FOG Reporting:
              https://fog-external-reporting-results.fogproject.us/

              M 1 Reply Last reply Jul 20, 2015, 6:47 PM Reply Quote 0
              • M
                Marcus Allen @Wayne Workman
                last edited by Jul 20, 2015, 6:47 PM

                @Wayne-Workman

                I dont think that is working right:

                du -a -h / | sort -n -r | head -n 10
                1020K   /usr/lib/kbd/consolefonts
                1016K   /usr/share/doc/sudo-1.8.6p7
                1016K   /usr/lib64/python2.7/site-packages/gi
                1012K   /usr/share/pki/ca-trust-source
                1012K   /usr/share/pki
                1012K   /usr/lib/udev/hwdb.d/20-usb-vendor-model.hwdb
                1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko
                1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x
                1008K   /usr/lib/firmware/cxgb4
                1004K   /var/lib/mysql/mysql
                

                Is that re right spot to put that switch in?

                W 1 Reply Last reply Jul 20, 2015, 6:49 PM Reply Quote 0
                • W
                  Wayne Workman @Marcus Allen
                  last edited by Jul 20, 2015, 6:49 PM

                   du -a -h / | sort -n -r -h | head -n 10
                  

                  or this:

                   du -a -h / | sort -nrh | head -n 10 
                  

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                  Daily Clean Installation Results:
                  https://fogtesting.fogproject.us/
                  FOG Reporting:
                  https://fog-external-reporting-results.fogproject.us/

                  M 1 Reply Last reply Jul 20, 2015, 6:50 PM Reply Quote 0
                  • M
                    Marcus Allen @Wayne Workman
                    last edited by Jul 20, 2015, 6:50 PM

                    @Wayne-Workman said:

                    du -a -h / | sort -n -r -h | head -n 10

                    sort: options '-hn' are incompatible
                    
                    W 1 Reply Last reply Jul 20, 2015, 6:54 PM Reply Quote 0
                    • W
                      Wayne Workman @Marcus Allen
                      last edited by Jul 20, 2015, 6:54 PM

                      @Marcus-Allen Ok. Your last post you put up was in human readable format so do it that way.

                      Can you post this?

                      df -h
                      

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                      Daily Clean Installation Results:
                      https://fogtesting.fogproject.us/
                      FOG Reporting:
                      https://fog-external-reporting-results.fogproject.us/

                      1 Reply Last reply Reply Quote 0
                      • M
                        Marcus Allen
                        last edited by Jul 20, 2015, 6:55 PM

                        Filesystem               Size  Used Avail Use% Mounted on
                        /dev/mapper/centos-root   30G   30G  303M 100% /
                        devtmpfs                 911M     0  911M   0% /dev
                        tmpfs                    921M     0  921M   0% /dev/shm
                        tmpfs                    921M  8.5M  912M   1% /run
                        tmpfs                    921M     0  921M   0% /sys/fs/cgroup
                        /dev/sda1                497M  120M  378M  24% /boot
                        
                        
                        W 1 Reply Last reply Jul 20, 2015, 6:56 PM Reply Quote 0
                        • W
                          Wayne Workman @Marcus Allen
                          last edited by Jul 20, 2015, 6:56 PM

                          @Marcus-Allen

                          Is that correct? you basically only have 30 GB available on this server?

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                          Daily Clean Installation Results:
                          https://fogtesting.fogproject.us/
                          FOG Reporting:
                          https://fog-external-reporting-results.fogproject.us/

                          M 1 Reply Last reply Jul 20, 2015, 7:04 PM Reply Quote 0
                          • M
                            Marcus Allen
                            last edited by Jul 20, 2015, 6:57 PM

                            Thats correct. Its a VM, I can increase the disk space if needed, however last week I was using about 11GB, so I didnt bump it up yet.

                            W 1 Reply Last reply Jul 20, 2015, 7:04 PM Reply Quote 0
                            • M
                              Marcus Allen @Wayne Workman
                              last edited by Jul 20, 2015, 7:04 PM

                              @Wayne-Workman

                              I’m bumping up to 64gb. This is a FOG Only server, no images are stored here.

                              1 Reply Last reply Reply Quote 0
                              • W
                                Wayne Workman @Marcus Allen
                                last edited by Jul 20, 2015, 7:04 PM

                                @Marcus-Allen I just tried this, it worked better than the other one:

                                find . -type f -printf "%s\t%p\n" | sort -n | tail -10
                                

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                Daily Clean Installation Results:
                                https://fogtesting.fogproject.us/
                                FOG Reporting:
                                https://fog-external-reporting-results.fogproject.us/

                                M 1 Reply Last reply Jul 20, 2015, 7:07 PM Reply Quote 0
                                • M
                                  Marcus Allen @Wayne Workman
                                  last edited by Jul 20, 2015, 7:07 PM

                                  @Wayne-Workman

                                  15773912        ./var/www/html/fog/service/ipxe/init.xz
                                  15775828        ./var/www/html/fog.prev/service/ipxe/init.xz
                                  18102074        ./boot/initramfs-3.10.0-229.el7.x86_64.img
                                  18179668        ./boot/initramfs-3.10.0-229.el7.x86_64kdump.img
                                  18874368        ./var/lib/mysql/ibdata1
                                  24281088        ./var/lib/rpm/Packages
                                  39835970        ./boot/initramfs-0-rescue-e248d8e9ddcb47a99cdece8ee62fabc5.img
                                  67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1
                                  67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc
                                  140737486266368 ./proc/kcore
                                  
                                  
                                  W 2 Replies Last reply Jul 20, 2015, 7:08 PM Reply Quote 0
                                  • W
                                    Wayne Workman @Marcus Allen
                                    last edited by Jul 20, 2015, 7:08 PM

                                    @Marcus-Allen I think it’s obvious where your space went…

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                    Daily Clean Installation Results:
                                    https://fogtesting.fogproject.us/
                                    FOG Reporting:
                                    https://fog-external-reporting-results.fogproject.us/

                                    M 1 Reply Last reply Jul 20, 2015, 7:13 PM Reply Quote 0
                                    • M
                                      Marcus Allen @Wayne Workman
                                      last edited by Jul 20, 2015, 7:13 PM

                                      @Marcus-Allen said:

                                      proc/kcore

                                      Got it

                                      W 1 Reply Last reply Jul 20, 2015, 8:19 PM Reply Quote 0
                                      • W
                                        Wayne Workman @Marcus Allen
                                        last edited by Wayne Workman Jul 20, 2015, 2:19 PM Jul 20, 2015, 8:19 PM

                                        @Marcus-Allen Just following up on this thread, @Wolfbane8653 was correct (ofcourse).

                                        But, on Red-Hat based distros, the path to the logs is in

                                        /var/log/httpd

                                        I just had to deal with this issue at one of my sites… fog server had a 300GB Apache log file lol. I deleted the file and then restarted httpd. Some of the tables in the fog DB were marked as crashed as well. I had to issue a repair statement on those to fix them like this:

                                        mysqlcheck --repair --databases fog
                                        

                                        and then I gave mysql a restart and all was fine.

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                        Daily Clean Installation Results:
                                        https://fogtesting.fogproject.us/
                                        FOG Reporting:
                                        https://fog-external-reporting-results.fogproject.us/

                                        M 1 Reply Last reply Jul 20, 2015, 8:25 PM Reply Quote 0
                                        • M
                                          Marcus Allen @Wayne Workman
                                          last edited by Jul 20, 2015, 8:25 PM

                                          @Wayne-Workman said:

                                          Sorry, I’m a touch confused now. My logs are empty in the /var/log/httpd directory. I even cleaned them out too just to be safe, still only about 10MB.

                                          Or, were you just mentioning about the log files in general?

                                          W 2 Replies Last reply Jul 20, 2015, 9:16 PM Reply Quote 0
                                          • 1
                                          • 2
                                          • 1 / 2
                                          1 / 2
                                          • First post
                                            3/23
                                            Last post

                                          195

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project