Fog server keeps going down

[szecca1]

@Wayne-Workman Do you want me to do this?

[szecca1]

@Tom-Elliott Yes we tried entering something and nothing is returned at all

[szecca1]

@szecca1 I just walked in today and the server is not reachable. I figured before I reboot it I would see if you guys want me to do anything. I can ping the server and it responds but trying to putty into the server it just hangs and never connects.

[Wayne Workman]

@szecca1 Can you get to it through the hypervisor?

[szecca1]

@Wayne-Workman I apologize I was trying to get a hold of my boss to log onto vSphere. When I go to the console, it asks for localhost login and then gives an error:

(38484.938838) Out of memory: kill process 2739 (httpd) score 104 or sacrifice child
(38484.942704) killed process 2739 (httpd) total -vm:1247792kb, anon-rss: 412668kb, file-rss:0KB

[szecca1]

@Wayne-Workman I am holding off on rebooting for a little bit just so you guys can see that error message

[Wayne Workman]

@szecca1 It displays that error AT the login screen, or after?

[Wayne Workman]

@szecca1 Check the amount of free RAM.

free -m

http://www.cyberciti.biz/faq/linux-check-memory-usage/

What does top say for memory usage in httpd?

[szecca1]

@Wayne-Workman as soon as the localhost login screen comes up, I click and that error message comes up.

[Wayne Workman]

@szecca1 said:

@Wayne-Workman I apologize I was trying to get a hold of my boss to log onto vSphere. When I go to the console, it asks for localhost login and then gives an error:

(38484.938838) Out of memory: kill process 2739 (httpd) score 104 or sacrifice child
(38484.942704) killed process 2739 (httpd) total -vm:1247792kb, anon-rss: 412668kb, file-rss:0KB

1247792KB + 412668KB = 1660460KB = 1660.46MB = 1.24GB

How much RAM does the server have?

[szecca1]

@Wayne-Workman I can’t type anything in as the screen wont allow me to even log into the machine until I reboot it

[szecca1]

@Wayne-Workman We set up the server with 4096 MB

[szecca1]

@Wayne-Workman But it says the memory overhead is 88MB but not sure what that means

[Wayne Workman]

@szecca1 So it’s simple… the server ran out of RAM - or - has dynamic RAM assigned and VMWare tried to smoosh it down a little to make room for other VMs that have a need… I think.

I was just reading this: https://plumbr.eu/blog/memory-leaks/out-of-memory-kill-process-or-sacrifice-child

This seems to say the same thing: http://ubuntuforums.org/showthread.php?t=2070853

[szecca1]

@Wayne-Workman So you would recommend adding more Ram? What do you think I should up it to?

[Wayne Workman]

@szecca1 I couldn’t see how it’s actually using all of 4 gigs… Make sure in VMWare that fog’s ram is not set to dynamic (auto expanding and contracting).

if it is set to dynamic, you could push the memory weight higher (it’s priority).

You could try bumping it to 6 gigs… but I really really doubt that the 4 gigs it has is being used… I really think dynamic RAM is to blame.

[szecca1]

@Wayne-Workman How do I see if it is set to dynamic. All I am seeing is the amount of ram nothing saying dynamic or not!

[Wayne Workman]

@szecca1 I have no idea, I don’t have any VMWare experience… but probably the other @Developers and @Moderators could chime in on that.

[szecca1]

@Wayne-Workman Better question, why would this randomly just start happening. The server was working fine until recently

[Wayne Workman]

@szecca1 Ask your boss if a new VM has been created on that physical box. Ask if any changes on other VMs on that box happened around the time that the FOG server started having issues.

And maybe just bump it to 6 gigs and see what happens tomorrow morning.