• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Fog server keeps going down

    Scheduled Pinned Locked Moved Solved
    FOG Problems
    7
    103
    85.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      szecca1 @Wayne Workman
      last edited by Wayne Workman

      @Wayne-Workman I apologize I was trying to get a hold of my boss to log onto vSphere. When I go to the console, it asks for localhost login and then gives an error:

      (38484.938838) Out of memory: kill process 2739 (httpd) score 104 or sacrifice child
      (38484.942704) killed process 2739 (httpd) total -vm:1247792kb, anon-rss: 412668kb, file-rss:0KB
      
      Wayne WorkmanW 1 Reply Last reply Reply Quote 0
      • S
        szecca1 @Wayne Workman
        last edited by

        @Wayne-Workman I am holding off on rebooting for a little bit just so you guys can see that error message

        Wayne WorkmanW 2 Replies Last reply Reply Quote 0
        • Wayne WorkmanW
          Wayne Workman @szecca1
          last edited by

          @szecca1 It displays that error AT the login screen, or after?

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
          Daily Clean Installation Results:
          https://fogtesting.fogproject.us/
          FOG Reporting:
          https://fog-external-reporting-results.fogproject.us/

          S 1 Reply Last reply Reply Quote 0
          • Wayne WorkmanW
            Wayne Workman @szecca1
            last edited by Wayne Workman

            @szecca1 Check the amount of free RAM.

            free -m

            http://www.cyberciti.biz/faq/linux-check-memory-usage/

            What does top say for memory usage in httpd?

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
            Daily Clean Installation Results:
            https://fogtesting.fogproject.us/
            FOG Reporting:
            https://fog-external-reporting-results.fogproject.us/

            S 1 Reply Last reply Reply Quote 0
            • S
              szecca1 @Wayne Workman
              last edited by

              @Wayne-Workman as soon as the localhost login screen comes up, I click and that error message comes up.

              1 Reply Last reply Reply Quote 0
              • Wayne WorkmanW
                Wayne Workman @szecca1
                last edited by Wayne Workman

                @szecca1 said:

                @Wayne-Workman I apologize I was trying to get a hold of my boss to log onto vSphere. When I go to the console, it asks for localhost login and then gives an error:

                (38484.938838) Out of memory: kill process 2739 (httpd) score 104 or sacrifice child
                (38484.942704) killed process 2739 (httpd) total -vm:1247792kb, anon-rss: 412668kb, file-rss:0KB
                

                1247792KB + 412668KB = 1660460KB = 1660.46MB = 1.24GB

                How much RAM does the server have?

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                Daily Clean Installation Results:
                https://fogtesting.fogproject.us/
                FOG Reporting:
                https://fog-external-reporting-results.fogproject.us/

                S 2 Replies Last reply Reply Quote 0
                • S
                  szecca1 @Wayne Workman
                  last edited by

                  @Wayne-Workman I can’t type anything in as the screen wont allow me to even log into the machine until I reboot it

                  Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                  • S
                    szecca1 @Wayne Workman
                    last edited by

                    @Wayne-Workman We set up the server with 4096 MB

                    1 Reply Last reply Reply Quote 0
                    • S
                      szecca1 @Wayne Workman
                      last edited by

                      @Wayne-Workman But it says the memory overhead is 88MB but not sure what that means

                      1 Reply Last reply Reply Quote 0
                      • Wayne WorkmanW
                        Wayne Workman @szecca1
                        last edited by Wayne Workman

                        @szecca1 So it’s simple… the server ran out of RAM - or - has dynamic RAM assigned and VMWare tried to smoosh it down a little to make room for other VMs that have a need… I think.

                        I was just reading this: https://plumbr.eu/blog/memory-leaks/out-of-memory-kill-process-or-sacrifice-child

                        This seems to say the same thing: http://ubuntuforums.org/showthread.php?t=2070853

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                        Daily Clean Installation Results:
                        https://fogtesting.fogproject.us/
                        FOG Reporting:
                        https://fog-external-reporting-results.fogproject.us/

                        S 1 Reply Last reply Reply Quote 0
                        • S
                          szecca1 @Wayne Workman
                          last edited by

                          @Wayne-Workman So you would recommend adding more Ram? What do you think I should up it to?

                          Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                          • Wayne WorkmanW
                            Wayne Workman @szecca1
                            last edited by

                            @szecca1 I couldn’t see how it’s actually using all of 4 gigs… Make sure in VMWare that fog’s ram is not set to dynamic (auto expanding and contracting).

                            if it is set to dynamic, you could push the memory weight higher (it’s priority).

                            You could try bumping it to 6 gigs… but I really really doubt that the 4 gigs it has is being used… I really think dynamic RAM is to blame.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                            Daily Clean Installation Results:
                            https://fogtesting.fogproject.us/
                            FOG Reporting:
                            https://fog-external-reporting-results.fogproject.us/

                            S 1 Reply Last reply Reply Quote 0
                            • S
                              szecca1 @Wayne Workman
                              last edited by

                              @Wayne-Workman How do I see if it is set to dynamic. All I am seeing is the amount of ram nothing saying dynamic or not!

                              Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                              • Wayne WorkmanW
                                Wayne Workman @szecca1
                                last edited by

                                @szecca1 I have no idea, I don’t have any VMWare experience… but probably the other @Developers and @Moderators could chime in on that.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                Daily Clean Installation Results:
                                https://fogtesting.fogproject.us/
                                FOG Reporting:
                                https://fog-external-reporting-results.fogproject.us/

                                S 1 Reply Last reply Reply Quote 0
                                • S
                                  szecca1 @Wayne Workman
                                  last edited by

                                  @Wayne-Workman Better question, why would this randomly just start happening. The server was working fine until recently

                                  Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                                  • Wayne WorkmanW
                                    Wayne Workman @szecca1
                                    last edited by

                                    @szecca1 Ask your boss if a new VM has been created on that physical box. Ask if any changes on other VMs on that box happened around the time that the FOG server started having issues.

                                    And maybe just bump it to 6 gigs and see what happens tomorrow morning.

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                    Daily Clean Installation Results:
                                    https://fogtesting.fogproject.us/
                                    FOG Reporting:
                                    https://fog-external-reporting-results.fogproject.us/

                                    S 1 Reply Last reply Reply Quote 0
                                    • S
                                      szecca1 @Wayne Workman
                                      last edited by

                                      @Wayne-Workman Ok i extended it to 6 gb and will update you tomorrow on if the server continues to do the same thing. My boss is out today so tomorrow I will discuss with him about any changes made and about dynamic RAM.

                                      Wayne WorkmanW 1 Reply Last reply Reply Quote 0
                                      • Wayne WorkmanW
                                        Wayne Workman @szecca1
                                        last edited by

                                        @szecca1 Can you please setup this crontab task? I wrote a script to monitor resource usage over time. This can prove invaluable in finding out WHEN and WHY the server is crashing.

                                        dt="$(date)"
                                        echo ----------------------------- >> /root/monitor.log
                                        echo $dt >> /root/monitor.log
                                        echo ----------------------------- >> /root/monitor.log
                                        /usr/bin/free -m >> /root/monitor.log
                                        echo ----------------------------- >> /root/monitor.log
                                        /usr/bin/top -n 1 >> /root/monitor.log
                                        echo ----------------------------- >> /root/monitor.log
                                        echo .............................. >> /root/monitor.log
                                        echo .............................. >> /root/monitor.log
                                        echo .............................. >> /root/monitor.log
                                        

                                        I put that script here:
                                        /root/monitor.sh

                                        Made it executable:
                                        chmod +x /root/monitor.sh

                                        Then added a crontab entry to run it every 15 minutes:
                                        crontab -e

                                        0,15,30,45 * * * * /root/monitor.sh
                                        

                                        Your output should look something like this:
                                        cat /root/monitor.sh

                                        -----------------------------
                                        Mon Nov 30 12:42:53 CST 2015
                                        -----------------------------
                                                      total        used        free      shared  buff/cache   available
                                        Mem:           3934         409        2066           0        1458        3197
                                        Swap:          2047           0        2047
                                        -----------------------------
                                        top - 12:42:53 up 29 min,  1 user,  load average: 3.68, 4.22, 3.76
                                        Tasks: 180 total,   3 running, 177 sleeping,   0 stopped,   0 zombie
                                        %Cpu(s): 29.8 us, 37.6 sy,  0.0 ni, 31.5 id,  0.2 wa,  0.0 hi,  0.9 si,  0.0 st
                                        KiB Mem :  4028768 total,  2123932 free,   411532 used,  1493304 buff/cache
                                        KiB Swap:  2097148 total,  2097148 free,        0 used.  3281768 avail Mem
                                        
                                          PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                                          559 root      20   0   49864  10204   4808 S  50.0  0.3  14:39.07 systemd-udevd
                                         2423 root      20   0       0      0      0 D  18.8  0.0   2:08.52 kworker/0:2
                                        55794 root      20   0   49864   8820   3420 R  18.8  0.2   0:00.03 systemd-udevd
                                        55793 root      20   0   49864   8820   3420 S  12.5  0.2   0:00.02 systemd-udevd
                                           35 root      20   0       0      0      0 R   6.2  0.0   1:25.50 kdevtmpfs
                                        55783 root      20   0  160796   4376   3688 R   6.2  0.1   0:00.01 top
                                        55795 root      20   0   49864   7324   1924 S   6.2  0.2   0:00.01 systemd-udevd
                                        55797 root      20   0   49864   7324   1924 S   6.2  0.2   0:00.01 systemd-udevd
                                            1 root      20   0   54716   8416   5696 S   0.0  0.2   0:02.41 systemd
                                            2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
                                            3 root      20   0       0      0      0 S   0.0  0.0   0:01.21 ksoftirqd/0
                                            5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
                                            7 root      20   0       0      0      0 S   0.0  0.0   0:16.32 rcu_sched
                                            8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
                                            9 root      20   0       0      0      0 S   0.0  0.0   0:17.35 rcuos/0
                                           10 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0
                                           11 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0
                                           12 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/0
                                           13 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/1
                                           14 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/1
                                           15 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/1
                                           17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
                                           18 root      20   0       0      0      0 S   0.0  0.0   0:05.64 rcuos/1
                                           19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1
                                           20 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/2
                                           21 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/2
                                           22 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/2
                                           24 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
                                           25 root      20   0       0      0      0 S   0.0  0.0   0:05.83 rcuos/2
                                           26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2
                                           27 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/3
                                           28 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/3
                                           29 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/3
                                           31 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
                                           32 root      20   0       0      0      0 S   0.0  0.0   0:06.48 rcuos/3
                                           33 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/3
                                           34 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 khelper
                                           36 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
                                           37 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 perf
                                           38 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback
                                           39 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd
                                           40 root      39  19       0      0      0 S   0.0  0.0   0:00.00 khugepaged
                                           41 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto
                                           42 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd
                                           43 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
                                        -----------------------------
                                        ..............................
                                        ..............................
                                        ..............................
                                        

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                        Daily Clean Installation Results:
                                        https://fogtesting.fogproject.us/
                                        FOG Reporting:
                                        https://fog-external-reporting-results.fogproject.us/

                                        S 1 Reply Last reply Reply Quote 1
                                        • cmlC
                                          cml Moderator
                                          last edited by cml

                                          You can set memory allocation in VMWare, in the VM properties go to the resources tab, and click on memory.

                                          You can either reserve all memory, or a set amount.

                                          0_1448972464611_2015-12-01 06_05_14-Fog server keeps going down _ FOG Project.png

                                          S 1 Reply Last reply Reply Quote 0
                                          • S
                                            szecca1 @Wayne Workman
                                            last edited by

                                            @Wayne-Workman I just got into work today and this morning the fog server was still up and running after the memory upgrade to 6 GBs. I can still do that if you’d like because I would hate to see your hard word go to waste. Let me know if you want me to do that

                                            Wayne WorkmanW 2 Replies Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 5
                                            • 6
                                            • 4 / 6
                                            • First post
                                              Last post

                                            217

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project