• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

Fog server keeps going down

Scheduled Pinned Locked Moved Solved
FOG Problems
7
103
82.8k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    szecca1 @Wayne Workman
    last edited by Wayne Workman Nov 30, 2015, 10:37 AM Nov 30, 2015, 4:32 PM

    @Wayne-Workman I apologize I was trying to get a hold of my boss to log onto vSphere. When I go to the console, it asks for localhost login and then gives an error:

    (38484.938838) Out of memory: kill process 2739 (httpd) score 104 or sacrifice child
    (38484.942704) killed process 2739 (httpd) total -vm:1247792kb, anon-rss: 412668kb, file-rss:0KB
    
    W 1 Reply Last reply Nov 30, 2015, 4:44 PM Reply Quote 0
    • S
      szecca1 @Wayne Workman
      last edited by Nov 30, 2015, 4:33 PM

      @Wayne-Workman I am holding off on rebooting for a little bit just so you guys can see that error message

      W 2 Replies Last reply Nov 30, 2015, 4:34 PM Reply Quote 0
      • W
        Wayne Workman @szecca1
        last edited by Nov 30, 2015, 4:34 PM

        @szecca1 It displays that error AT the login screen, or after?

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
        Daily Clean Installation Results:
        https://fogtesting.fogproject.us/
        FOG Reporting:
        https://fog-external-reporting-results.fogproject.us/

        S 1 Reply Last reply Nov 30, 2015, 4:43 PM Reply Quote 0
        • W
          Wayne Workman @szecca1
          last edited by Wayne Workman Nov 30, 2015, 10:36 AM Nov 30, 2015, 4:35 PM

          @szecca1 Check the amount of free RAM.

          free -m

          http://www.cyberciti.biz/faq/linux-check-memory-usage/

          What does top say for memory usage in httpd?

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
          Daily Clean Installation Results:
          https://fogtesting.fogproject.us/
          FOG Reporting:
          https://fog-external-reporting-results.fogproject.us/

          S 1 Reply Last reply Nov 30, 2015, 4:47 PM Reply Quote 0
          • S
            szecca1 @Wayne Workman
            last edited by Nov 30, 2015, 4:43 PM

            @Wayne-Workman as soon as the localhost login screen comes up, I click and that error message comes up.

            1 Reply Last reply Reply Quote 0
            • W
              Wayne Workman @szecca1
              last edited by Wayne Workman Nov 30, 2015, 10:44 AM Nov 30, 2015, 4:44 PM

              @szecca1 said:

              @Wayne-Workman I apologize I was trying to get a hold of my boss to log onto vSphere. When I go to the console, it asks for localhost login and then gives an error:

              (38484.938838) Out of memory: kill process 2739 (httpd) score 104 or sacrifice child
              (38484.942704) killed process 2739 (httpd) total -vm:1247792kb, anon-rss: 412668kb, file-rss:0KB
              

              1247792KB + 412668KB = 1660460KB = 1660.46MB = 1.24GB

              How much RAM does the server have?

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
              Daily Clean Installation Results:
              https://fogtesting.fogproject.us/
              FOG Reporting:
              https://fog-external-reporting-results.fogproject.us/

              S 2 Replies Last reply Nov 30, 2015, 4:48 PM Reply Quote 0
              • S
                szecca1 @Wayne Workman
                last edited by Nov 30, 2015, 4:47 PM

                @Wayne-Workman I can’t type anything in as the screen wont allow me to even log into the machine until I reboot it

                W 1 Reply Last reply Nov 30, 2015, 4:48 PM Reply Quote 0
                • S
                  szecca1 @Wayne Workman
                  last edited by Nov 30, 2015, 4:48 PM

                  @Wayne-Workman We set up the server with 4096 MB

                  1 Reply Last reply Reply Quote 0
                  • S
                    szecca1 @Wayne Workman
                    last edited by Nov 30, 2015, 4:48 PM

                    @Wayne-Workman But it says the memory overhead is 88MB but not sure what that means

                    1 Reply Last reply Reply Quote 0
                    • W
                      Wayne Workman @szecca1
                      last edited by Wayne Workman Nov 30, 2015, 10:49 AM Nov 30, 2015, 4:48 PM

                      @szecca1 So it’s simple… the server ran out of RAM - or - has dynamic RAM assigned and VMWare tried to smoosh it down a little to make room for other VMs that have a need… I think.

                      I was just reading this: https://plumbr.eu/blog/memory-leaks/out-of-memory-kill-process-or-sacrifice-child

                      This seems to say the same thing: http://ubuntuforums.org/showthread.php?t=2070853

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                      Daily Clean Installation Results:
                      https://fogtesting.fogproject.us/
                      FOG Reporting:
                      https://fog-external-reporting-results.fogproject.us/

                      S 1 Reply Last reply Nov 30, 2015, 4:50 PM Reply Quote 0
                      • S
                        szecca1 @Wayne Workman
                        last edited by Nov 30, 2015, 4:50 PM

                        @Wayne-Workman So you would recommend adding more Ram? What do you think I should up it to?

                        W 1 Reply Last reply Nov 30, 2015, 4:53 PM Reply Quote 0
                        • W
                          Wayne Workman @szecca1
                          last edited by Nov 30, 2015, 4:53 PM

                          @szecca1 I couldn’t see how it’s actually using all of 4 gigs… Make sure in VMWare that fog’s ram is not set to dynamic (auto expanding and contracting).

                          if it is set to dynamic, you could push the memory weight higher (it’s priority).

                          You could try bumping it to 6 gigs… but I really really doubt that the 4 gigs it has is being used… I really think dynamic RAM is to blame.

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                          Daily Clean Installation Results:
                          https://fogtesting.fogproject.us/
                          FOG Reporting:
                          https://fog-external-reporting-results.fogproject.us/

                          S 1 Reply Last reply Nov 30, 2015, 4:58 PM Reply Quote 0
                          • S
                            szecca1 @Wayne Workman
                            last edited by Nov 30, 2015, 4:58 PM

                            @Wayne-Workman How do I see if it is set to dynamic. All I am seeing is the amount of ram nothing saying dynamic or not!

                            W 1 Reply Last reply Nov 30, 2015, 5:00 PM Reply Quote 0
                            • W
                              Wayne Workman @szecca1
                              last edited by Nov 30, 2015, 5:00 PM

                              @szecca1 I have no idea, I don’t have any VMWare experience… but probably the other @Developers and @Moderators could chime in on that.

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                              Daily Clean Installation Results:
                              https://fogtesting.fogproject.us/
                              FOG Reporting:
                              https://fog-external-reporting-results.fogproject.us/

                              S 1 Reply Last reply Nov 30, 2015, 5:04 PM Reply Quote 0
                              • S
                                szecca1 @Wayne Workman
                                last edited by Nov 30, 2015, 5:04 PM

                                @Wayne-Workman Better question, why would this randomly just start happening. The server was working fine until recently

                                W 1 Reply Last reply Nov 30, 2015, 5:06 PM Reply Quote 0
                                • W
                                  Wayne Workman @szecca1
                                  last edited by Nov 30, 2015, 5:06 PM

                                  @szecca1 Ask your boss if a new VM has been created on that physical box. Ask if any changes on other VMs on that box happened around the time that the FOG server started having issues.

                                  And maybe just bump it to 6 gigs and see what happens tomorrow morning.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                  Daily Clean Installation Results:
                                  https://fogtesting.fogproject.us/
                                  FOG Reporting:
                                  https://fog-external-reporting-results.fogproject.us/

                                  S 1 Reply Last reply Nov 30, 2015, 5:41 PM Reply Quote 0
                                  • S
                                    szecca1 @Wayne Workman
                                    last edited by Nov 30, 2015, 5:41 PM

                                    @Wayne-Workman Ok i extended it to 6 gb and will update you tomorrow on if the server continues to do the same thing. My boss is out today so tomorrow I will discuss with him about any changes made and about dynamic RAM.

                                    W 1 Reply Last reply Nov 30, 2015, 6:44 PM Reply Quote 0
                                    • W
                                      Wayne Workman @szecca1
                                      last edited by Nov 30, 2015, 6:44 PM

                                      @szecca1 Can you please setup this crontab task? I wrote a script to monitor resource usage over time. This can prove invaluable in finding out WHEN and WHY the server is crashing.

                                      dt="$(date)"
                                      echo ----------------------------- >> /root/monitor.log
                                      echo $dt >> /root/monitor.log
                                      echo ----------------------------- >> /root/monitor.log
                                      /usr/bin/free -m >> /root/monitor.log
                                      echo ----------------------------- >> /root/monitor.log
                                      /usr/bin/top -n 1 >> /root/monitor.log
                                      echo ----------------------------- >> /root/monitor.log
                                      echo .............................. >> /root/monitor.log
                                      echo .............................. >> /root/monitor.log
                                      echo .............................. >> /root/monitor.log
                                      

                                      I put that script here:
                                      /root/monitor.sh

                                      Made it executable:
                                      chmod +x /root/monitor.sh

                                      Then added a crontab entry to run it every 15 minutes:
                                      crontab -e

                                      0,15,30,45 * * * * /root/monitor.sh
                                      

                                      Your output should look something like this:
                                      cat /root/monitor.sh

                                      -----------------------------
                                      Mon Nov 30 12:42:53 CST 2015
                                      -----------------------------
                                                    total        used        free      shared  buff/cache   available
                                      Mem:           3934         409        2066           0        1458        3197
                                      Swap:          2047           0        2047
                                      -----------------------------
                                      top - 12:42:53 up 29 min,  1 user,  load average: 3.68, 4.22, 3.76
                                      Tasks: 180 total,   3 running, 177 sleeping,   0 stopped,   0 zombie
                                      %Cpu(s): 29.8 us, 37.6 sy,  0.0 ni, 31.5 id,  0.2 wa,  0.0 hi,  0.9 si,  0.0 st
                                      KiB Mem :  4028768 total,  2123932 free,   411532 used,  1493304 buff/cache
                                      KiB Swap:  2097148 total,  2097148 free,        0 used.  3281768 avail Mem
                                      
                                        PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                                        559 root      20   0   49864  10204   4808 S  50.0  0.3  14:39.07 systemd-udevd
                                       2423 root      20   0       0      0      0 D  18.8  0.0   2:08.52 kworker/0:2
                                      55794 root      20   0   49864   8820   3420 R  18.8  0.2   0:00.03 systemd-udevd
                                      55793 root      20   0   49864   8820   3420 S  12.5  0.2   0:00.02 systemd-udevd
                                         35 root      20   0       0      0      0 R   6.2  0.0   1:25.50 kdevtmpfs
                                      55783 root      20   0  160796   4376   3688 R   6.2  0.1   0:00.01 top
                                      55795 root      20   0   49864   7324   1924 S   6.2  0.2   0:00.01 systemd-udevd
                                      55797 root      20   0   49864   7324   1924 S   6.2  0.2   0:00.01 systemd-udevd
                                          1 root      20   0   54716   8416   5696 S   0.0  0.2   0:02.41 systemd
                                          2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
                                          3 root      20   0       0      0      0 S   0.0  0.0   0:01.21 ksoftirqd/0
                                          5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
                                          7 root      20   0       0      0      0 S   0.0  0.0   0:16.32 rcu_sched
                                          8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
                                          9 root      20   0       0      0      0 S   0.0  0.0   0:17.35 rcuos/0
                                         10 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0
                                         11 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0
                                         12 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/0
                                         13 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/1
                                         14 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/1
                                         15 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/1
                                         17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
                                         18 root      20   0       0      0      0 S   0.0  0.0   0:05.64 rcuos/1
                                         19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1
                                         20 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/2
                                         21 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/2
                                         22 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/2
                                         24 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
                                         25 root      20   0       0      0      0 S   0.0  0.0   0:05.83 rcuos/2
                                         26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2
                                         27 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/3
                                         28 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/3
                                         29 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/3
                                         31 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
                                         32 root      20   0       0      0      0 S   0.0  0.0   0:06.48 rcuos/3
                                         33 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/3
                                         34 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 khelper
                                         36 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
                                         37 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 perf
                                         38 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback
                                         39 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd
                                         40 root      39  19       0      0      0 S   0.0  0.0   0:00.00 khugepaged
                                         41 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto
                                         42 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd
                                         43 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
                                      -----------------------------
                                      ..............................
                                      ..............................
                                      ..............................
                                      

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                      Daily Clean Installation Results:
                                      https://fogtesting.fogproject.us/
                                      FOG Reporting:
                                      https://fog-external-reporting-results.fogproject.us/

                                      S 1 Reply Last reply Dec 1, 2015, 1:21 PM Reply Quote 1
                                      • cmlC
                                        cml Moderator
                                        last edited by cml Dec 1, 2015, 6:21 AM Dec 1, 2015, 12:20 PM

                                        You can set memory allocation in VMWare, in the VM properties go to the resources tab, and click on memory.

                                        You can either reserve all memory, or a set amount.

                                        0_1448972464611_2015-12-01 06_05_14-Fog server keeps going down _ FOG Project.png

                                        S 1 Reply Last reply Dec 1, 2015, 1:22 PM Reply Quote 0
                                        • S
                                          szecca1 @Wayne Workman
                                          last edited by Dec 1, 2015, 1:21 PM

                                          @Wayne-Workman I just got into work today and this morning the fog server was still up and running after the memory upgrade to 6 GBs. I can still do that if you’d like because I would hate to see your hard word go to waste. Let me know if you want me to do that

                                          W 2 Replies Last reply Dec 1, 2015, 1:24 PM Reply Quote 0
                                          • 1
                                          • 2
                                          • 3
                                          • 4
                                          • 5
                                          • 6
                                          • 4 / 6
                                          4 / 6
                                          • First post
                                            62/103
                                            Last post

                                          147

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project