Fog server keeps going down
-
@szecca1 Can you please setup this crontab task? I wrote a script to monitor resource usage over time. This can prove invaluable in finding out WHEN and WHY the server is crashing.
dt="$(date)" echo ----------------------------- >> /root/monitor.log echo $dt >> /root/monitor.log echo ----------------------------- >> /root/monitor.log /usr/bin/free -m >> /root/monitor.log echo ----------------------------- >> /root/monitor.log /usr/bin/top -n 1 >> /root/monitor.log echo ----------------------------- >> /root/monitor.log echo .............................. >> /root/monitor.log echo .............................. >> /root/monitor.log echo .............................. >> /root/monitor.logI put that script here:
/root/monitor.shMade it executable:
chmod +x /root/monitor.shThen added a crontab entry to run it every 15 minutes:
crontab -e0,15,30,45 * * * * /root/monitor.shYour output should look something like this:
cat /root/monitor.sh----------------------------- Mon Nov 30 12:42:53 CST 2015 ----------------------------- total used free shared buff/cache available Mem: 3934 409 2066 0 1458 3197 Swap: 2047 0 2047 ----------------------------- top - 12:42:53 up 29 min, 1 user, load average: 3.68, 4.22, 3.76 Tasks: 180 total, 3 running, 177 sleeping, 0 stopped, 0 zombie %Cpu(s): 29.8 us, 37.6 sy, 0.0 ni, 31.5 id, 0.2 wa, 0.0 hi, 0.9 si, 0.0 st KiB Mem : 4028768 total, 2123932 free, 411532 used, 1493304 buff/cache KiB Swap: 2097148 total, 2097148 free, 0 used. 3281768 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 559 root 20 0 49864 10204 4808 S 50.0 0.3 14:39.07 systemd-udevd 2423 root 20 0 0 0 0 D 18.8 0.0 2:08.52 kworker/0:2 55794 root 20 0 49864 8820 3420 R 18.8 0.2 0:00.03 systemd-udevd 55793 root 20 0 49864 8820 3420 S 12.5 0.2 0:00.02 systemd-udevd 35 root 20 0 0 0 0 R 6.2 0.0 1:25.50 kdevtmpfs 55783 root 20 0 160796 4376 3688 R 6.2 0.1 0:00.01 top 55795 root 20 0 49864 7324 1924 S 6.2 0.2 0:00.01 systemd-udevd 55797 root 20 0 49864 7324 1924 S 6.2 0.2 0:00.01 systemd-udevd 1 root 20 0 54716 8416 5696 S 0.0 0.2 0:02.41 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:01.21 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 7 root 20 0 0 0 0 S 0.0 0.0 0:16.32 rcu_sched 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 9 root 20 0 0 0 0 S 0.0 0.0 0:17.35 rcuos/0 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0 11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 12 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 13 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 14 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/1 15 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/1 17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H 18 root 20 0 0 0 0 S 0.0 0.0 0:05.64 rcuos/1 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1 20 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 21 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/2 22 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/2 24 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H 25 root 20 0 0 0 0 S 0.0 0.0 0:05.83 rcuos/2 26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2 27 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 28 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/3 29 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/3 31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H 32 root 20 0 0 0 0 S 0.0 0.0 0:06.48 rcuos/3 33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3 34 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper 36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns 37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 perf 38 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback 39 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd 40 root 39 19 0 0 0 S 0.0 0.0 0:00.00 khugepaged 41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto 42 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd 43 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset ----------------------------- .............................. .............................. .............................. -
You can set memory allocation in VMWare, in the VM properties go to the resources tab, and click on memory.
You can either reserve all memory, or a set amount.
-
@Wayne-Workman I just got into work today and this morning the fog server was still up and running after the memory upgrade to 6 GBs. I can still do that if you’d like because I would hate to see your hard word go to waste. Let me know if you want me to do that
-
@cml Thank you I saw this right before I upgraded the memory but didn’t know what to do and didn’t want to mess with something I didn’t understand. The memory upgrade seems to have fixed the issue for now and I will keep an eye on it but do you think I should do this anyways?
-
@cml Thank yo ufor the screen shot! I only have Hyper-V & KVM experience, glad someone could give a VMWare screenshot!
@szecca1 Yes please go ahead and setup the crontab event. Because the server is still online today - we know your issue is memory related, and the output from the below scheduled script will help you in the future… Keep in mind - the 6 gigs of RAM may not have fixed it, it may only have lengthened the amount of time between crashes…
-
This is what I am getting when I typed that path in. Am I doing something wrong?
-
@szecca1 The path is
/root/the filename ismonitor.shYou create the file. I made it from scratch. You can create it with Vi. Like
vi /root/monitor.shI suppose you can put the script anywhere you like.
-
@Wayne-Workman Hey after I am done typing the script how do I save it?
-
@Wayne-Workman I’m sorry I am having a difficult time doing this, so I’m just going to skip it unless you have more instructions on how to do this.
-
@szecca1 https://wiki.fogproject.org/wiki/index.php/Vi
Also - use putty and just copy / paste. No need to type all that out.
In Putty, you can copy by selecting the text and then right clicking anywhere on the window.
In Putty, you can paste by simply putting the cursor where you want your text to go, and then right click anywhere on the window. -
@Wayne-Workman Thank you but clearly I did something wrong. This is my output when I followed your instructions
-
@szecca1 The cat command just outputs the contents of a file for viewing.
You run executibles in Linux with
./Like
./installfog.sh?Try to run the script you just wrote. Like this:
cd /root
./monitor.sh
then output the contents of the log using thecatcommand.
cat /root/monitor.logYou can also look at the log using vi…
vi /root/monitor.log -
@Wayne-Workman I may have found my problem but not sure how to fix it
-
@szecca1 That’s not a big deal. What gets ran is what is stored on the disk.
Can you run the script? Is there output in the log?
-
@Wayne-Workman No I tried to see if the file was in the /root folder and its not so i wanted to make sure that wasn’t a problem. I’ll just recreate it again
-
@szecca1 You’re using Fedora right?
Did you create the crontab event as root? the
crontab -ecommand will create a cron event and run as the user who created the event, meaning, not created using root, no permission to the /root directory. -
@szecca1 I dont know what I am doing wrong but when I type /root then ls the file is not there and I continue to go back and resave it and still nothing
-
@szecca1 Switch to the root user like this:
su rootand then create the crontab event.Apologies for not saying that earlier. I do everything as root.
-
@Wayne-Workman apparently I don’t have the password for root
-
@Wayne-Workman I was just doing everything through sudo and my password
