HIGH CPU Fog Services after update r5029 v6759
-
@Tom-Elliott said:
@baggar11 as I understand it, this is still potentially due to starting the services before its dependencies have started. This is different from the pegged on update, I think.
Can you simply try reinstall and see if it’s still happening? Or did update always work, only reboot caused these issues?
For my upgrades, anything after 6753 would peg my cpu after a reboot. During and after the upgrade process using the install.sh, everything was fine. It’s only after the reboot that the services would max out the cpu. Probably the infinite loop issue as you wrote about below.
-
@baggar11 Okay, I’ve added a sleep time of 10 seconds, just in case of these situations where the int returned is 0. We must have at least 1 second sleeptime ( I believe ), and this just isn’t happening.
The only word of caution I can think of, now, is while this might help with CPU Cycles, it will not make the FOG Services actually work properly as from Ubuntu’s standpoint the service is already running. While I could, potentially, come up with a way to restart the services more appropriately, I don’t know where to start at the moment.
-
@Tom-Elliott said:
@baggar11 Okay, I’ve added a sleep time of 10 seconds, just in case of these situations where the int returned is 0. We must have at least 1 second sleeptime ( I believe ), and this just isn’t happening.
The only word of caution I can think of, now, is while this might help with CPU Cycles, it will not make the FOG Services actually work properly as from Ubuntu’s standpoint the service is already running. While I could, potentially, come up with a way to restart the services more appropriately, I don’t know where to start at the moment.
I’m not a coder so take this with a grain of salt. The following is a similar approach to another open source project I use.
Make a script to run as a cron job every 5 minutes. On 1st boot, the script essentially checks for things like MySQL and network being up, and if all checks out starts the FOG services. All other times it runs, it would really only check for FOG services running and then exit. heh…
Of course, it’s probably a waste of time as I assume systemd will probably take care of these things more intelligently than regular init and/or upstart. Which Fedora is already using and Ubuntu’s next LTS will be running.
I really appreciate you taking a look into these things Tom. FOG is a very cool project.
-
Like Baggar is saying. Thanks for taking your time to looking at these things.
Its just anoying that it is taking 100% CPU & not working properly.
Take your time but its probably a problem for all the ubuntu 14.04 - users after one update. -
@Tom-Elliott After update to r5054 this morning
Looks like it is fixed now but i have not rebooted any of them…
Servertop - 07:57:51 up 23:03, 3 users, load average: 134.00, 122.87, 122.12 Tasks: 351 total, 32 running, 312 sleeping, 0 stopped, 7 zombie %Cpu(s): 55.3 us, 42.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 2.7 si, 0.0 st KiB Mem: 4355692 total, 3461944 used, 893748 free, 69372 buffers KiB Swap: 1046524 total, 16896 used, 1029628 free. 2231712 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17801 mysql 20 0 386360 85396 9744 S 19.5 2.0 3:07.93 mysqld 20119 www-data 20 0 111272 19556 13116 R 11.3 0.4 0:04.27 apache2 21537 www-data 20 0 0 0 0 Z 9.9 0.0 0:01.06 apache2 21239 www-data 20 0 111188 19952 13344 R 8.6 0.5 0:02.65 apache2 19002 www-data 20 0 111044 19584 13116 R 7.0 0.4 0:08.43 apache2 18759 www-data 20 0 111296 19812 13092 R 6.6 0.5 0:13.16 apache2 19062 www-data 20 0 110988 19788 13752 R 6.6 0.5 0:07.04 apache2 20017 www-data 20 0 110996 18988 12572 S 6.6 0.4 0:04.30 apache2 18845 www-data 20 0 111300 19240 12688 R 6.3 0.4 0:13.16 apache2 18875 www-data 20 0 111256 20412 13732 R 6.3 0.5 0:12.60 apache2
Storage Node Master
top - 07:58:14 up 19:25, 2 users, load average: 0.01, 0.06, 0.12 Tasks: 201 total, 1 running, 195 sleeping, 0 stopped, 5 zombie %Cpu(s): 0.2 us, 0.2 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 1014980 total, 975276 used, 39704 free, 106172 buffers KiB Swap: 1037308 total, 22532 used, 1014776 free. 443116 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 4616 3552 2432 S 0.0 0.3 0:02.85 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:03.88 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 7 root 20 0 0 0 0 S 0.0 0.0 0:26.13 rcu_sched 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 9 root rt 0 0 0 0 S 0.0 0.0 0:00.02 migration/0 10 root rt 0 0 0 0 S 0.0 0.0 0:00.18 watchdog/0 11 root rt 0 0 0 0 S 0.0 0.0 0:00.20 watchdog/1 12 root rt 0 0 0 0 S 0.0 0.0 0:00.02 migration/1
Storage Node 2nd
top - 07:58:31 up 22:59, 3 users, load average: 0.01, 0.02, 0.05 Tasks: 227 total, 1 running, 225 sleeping, 0 stopped, 1 zombie %Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 7735460 total, 7466792 used, 268668 free, 69996 buffers KiB Swap: 7828476 total, 0 used, 7828476 free. 6881904 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7 root 20 0 0 0 0 S 0.3 0.0 0:29.70 rcu_sched 11185 fog 20 0 5572 2852 2352 R 0.3 0.0 0:02.77 top 1 root 20 0 4732 3844 2580 S 0.0 0.0 0:01.60 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:04.09 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 9 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/0 10 root rt 0 0 0 0 S 0.0 0.0 0:00.30 watchdog/0 11 root rt 0 0 0 0 S 0.0 0.0 0:00.32 watchdog/1 12 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/1
-
you are right. CPU is fixed indead … Till you boot
-
-
@boeleke What?
You updated and still seeing CPU load issues? See the service_lib script now implicitly defines a sleep time if one cannot be found otherwise.
-
@Tom-Elliott
thought i had the last version.
I just updated & rebooted. Now working fine for a couple of minutes.think its ok now.
-
Wanted to post another update. Updated to 6893 this morning. Removed my sleep and service stop commands from rc.local and rebooted. CPU seems to be in check again. Thanks for all your hard work!