HIGH CPU Fog Services after update r5029 v6759

baggar11 · Mar 16, 2016, 8:35 PM

Sorry guys! I have been rolling the dev version for so long without issue, I totally forgot about the startup services possibly being an issue. I’m all set now. Here’s what I did.

Ran this on each service

sudo update-rc.d FOG*service* disable

Then, as suggested, added these line to my rc.local

sleep 5
service FOG*service* start

Wayne Workman · Mar 17, 2016, 4:30 AM

@baggar11 Very nice.

Raymond Bell · Mar 17, 2016, 12:38 PM

@Tom-Elliott Upgrade to latest pull r5037
server is fine if i start the services one at a time

but both storage nodes have high CPU
and i have started them one at a time to check
as soon as i start one it jumps to like 98% and levels out at about 50%

top - 07:35:37 up 16:54,  2 users,  load average: 4.40, 2.95, 3.82
Tasks: 219 total,   5 running, 214 sleeping,   0 stopped,   0 zombie
%Cpu(s): 69.7 us, 30.1 sy,  0.0 ni,  0.1 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:   7735460 total,  1368652 used,  6366808 free,   107516 buffers
KiB Swap:  7828476 total,        0 used,  7828476 free.   824568 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8183 root      20   0   41920  20308  15328 R  91.6  0.3   1:38.30 FOGMulticastMan
 8212 root      20   0   41920  20692  15712 R  91.6  0.3   1:38.56 FOGSnapinReplic
 8239 root      20   0   41920  20584  15604 R  82.7  0.3   1:28.03 FOGPingHosts
 8226 root      20   0   41920  20132  15156 R  70.1  0.3   1:34.14 FOGTaskSchedule
 8197 root      20   0   41920  20140  15160 S  63.7  0.3   1:36.08 FOGImageReplica
    1 root      20   0    4612   3872   2616 S   0.0  0.1   0:01.52 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:03.83 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   0:27.75 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.09 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.10 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.15 ksoftirqd/1
   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   16 root      rt   0       0      0      0 S   0.0  0.0   0:00.10 watchdog/2
   17 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/2

Tom Elliott · Mar 17, 2016, 12:38 PM

@Raymond-Bell Just going to guess but the Storage nodes, currently, are unable to talk with the main FOG Server?

Raymond Bell · Mar 17, 2016, 12:45 PM

@Tom-Elliott said:

@Raymond-Bell Just going to guess but the Storage nodes, currently, are unable to talk with the main FOG Server?

Editing the rc.local file now will let you know after reboot

Raymond Bell · Mar 17, 2016, 1:09 PM

@Tom-Elliott
After editing rc.local and rebooting all i got to web sign in and got this

Database Schema Installer / Updater
But i did not get this during set-up to the latest SVN

I click install and get ’

Install/Upgrade Successful!

The following errors occured

Update ID: 1 - 0

Database Error:

Too many connections, Message: Check that database is running

Database SQL:

CREATE DATABASE fog

Update ID: 1 - 1

Database Error:

Too many connections, Message: Check that database is running

Database SQL:

CREATE TABLE `fog`.`groupMembers` ( `gmID` int(11) NOT NULL auto_increment, `gmHostID` int(11) NOT NULL, `gmGroupID` int(11) NOT NULL, PRIMARY KEY (`gmID`),
KEY `new_index` (`gmHostID`),
KEY `new_index1` (`gmGroupID`) ) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC

But second time login just fine and server is seeing nodes

After reboot and rc.local edit

Server

top - 07:58:10 up 8 min,  2 users,  load average: 127.49, 100.92, 50.49
Tasks: 360 total,  36 running, 324 sleeping,   0 stopped,   0 zombie
%Cpu(s): 65.6 us, 33.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  1.1 si,  0.0 st
KiB Mem:   4355692 total,  1476112 used,  2879580 free,    95208 buffers
KiB Swap:  1046524 total,        0 used,  1046524 free.   496208 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1363 root      20   0   39820  20000  15036 R  89.2  0.5   7:10.67 FOGSnapinReplic
 1399 root      20   0   39820  19996  15024 S  84.5  0.5   7:10.53 FOGTaskSchedule
 1254 mysql     20   0  363320  75940   9864 S  22.5  1.7   1:58.49 mysqld
 3441 www-data  20   0  108088  15672  10396 D   9.6  0.4   0:01.11 apache2
 3052 www-data  20   0  110464  18732  12840 R   8.9  0.4   0:06.03 apache2
 2983 www-data  20   0  110820  18828  12840 R   8.3  0.4   0:05.70 apache2
 3405 fog       20   0    5692   2916   2344 R   6.6  0.1   0:01.32 top
 3523 www-data  20   0  108232  15724  10312 R   6.6  0.4   0:00.72 apache2
 3060 www-data  20   0  111052  19272  12792 R   6.3  0.4   0:05.67 apache2
 3541 www-data  20   0  107928  15280  10012 R   5.6  0.4   0:00.45 apache2

SN 1

top - 07:59:33 up 9 min,  2 users,  load average: 5.07, 4.46, 2.47
Tasks: 221 total,   7 running, 214 sleeping,   0 stopped,   0 zombie
%Cpu(s): 69.8 us, 29.9 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem:   7735460 total,   881520 used,  6853940 free,    79604 buffers
KiB Swap:  7828476 total,        0 used,  7828476 free.   405380 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2643 root      20   0   39948  19864  14908 R  86.8  0.3   7:11.84 FOGImageReplica
 2679 root      20   0   39948  20068  15116 R  86.8  0.3   7:09.10 FOGMulticastMan
 2609 root      20   0   39948  19848  14888 R  76.8  0.3   7:09.03 FOGPingHosts
 2626 root      20   0   39948  19888  14928 R  74.8  0.3   7:09.72 FOGTaskSchedule
 2662 root      20   0   39948  19984  15024 R  74.1  0.3   7:11.40 FOGSnapinReplic
   41 root      20   0       0      0      0 R   0.3  0.0   0:00.17 kworker/0:1
 2845 fog       20   0    5544   2752   2344 R   0.3  0.0   0:00.35 top
    1 root      20   0    4740   3888   2616 S   0.0  0.1   0:01.44 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   0:00.29 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/0

SN 2

top - 08:00:12 up 10 min,  3 users,  load average: 5.12, 4.54, 2.58
Tasks: 225 total,   7 running, 218 sleeping,   0 stopped,   0 zombie
%Cpu(s): 72.2 us, 27.8 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   7735460 total,   886896 used,  6848564 free,    79612 buffers
KiB Swap:  7828476 total,        0 used,  7828476 free.   405392 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2662 root      20   0   39948  19984  15024 R  88.8  0.3   7:41.96 FOGSnapinReplic
 2643 root      20   0   39948  19864  14908 R  88.4  0.3   7:41.97 FOGImageReplica
 2626 root      20   0   39948  19888  14928 R  76.6  0.3   7:41.53 FOGTaskSchedule
 2609 root      20   0   39948  19848  14888 R  70.5  0.3   7:40.83 FOGPingHosts
 2679 root      20   0   39948  20068  15116 R  69.7  0.3   7:40.03 FOGMulticastMan
 2308 root      20   0   53452   6756   5804 S   5.7  0.1   0:00.33 udisksd
 2845 fog       20   0    5544   2876   2344 S   0.4  0.0   0:00.45 top
 2901 fog       20   0    5688   2768   2344 R   0.4  0.0   0:00.01 top
    1 root      20   0    4740   3888   2616 S   0.0  0.1   0:01.44 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   0:00.31 rcu_sched

Raymond Bell · Mar 17, 2016, 1:35 PM

Still don’t understand why i have so many apache2 services running on the server either…

Tom Elliott · Mar 17, 2016, 1:36 PM

@Raymond-Bell are you disabling the services from booting completely? Then using the rc.local?

THe rc.local won’t work if it can’t stop the original tasking anyway.

Raymond Bell · Mar 17, 2016, 1:37 PM

@Tom-Elliott said:

@Raymond-Bell are you disabling the services from booting completely? Then using the rc.local?

THe rc.local won’t work if it can’t stop the original tasking anyway.

Yes and i rebooted also

Quazz · Mar 17, 2016, 1:39 PM

I have 11 apache processes running right now, I’m fairly certain it’s normal behavior. Load is under 0.5% total.

Raymond Bell · Mar 17, 2016, 1:40 PM

@Wayne-Workman i will try this i think this is what Tom is talking about

Tom Elliott · Mar 17, 2016, 1:40 PM

@Raymond-Bell said:

Too many connections, Message: Check that database is running

Too many connections, Message: Check that database is running.

Can you edit the /etc/mysql/my.cnf file and add in the [mysqld] portion (or change) max_connections = 500 then restart the mysql service? (sudo service mysql restart)

Raymond Bell · Mar 17, 2016, 2:13 PM

@Tom-Elliott said:

@Raymond-Bell said:

Too many connections, Message: Check that database is running

Too many connections, Message: Check that database is running.

Can you edit the /etc/mysql/my.cnf file and add in the [mysqld] portion (or change) max_connections = 500 then restart the mysql service? (sudo service mysql restart)

Yes that helped with the to many connections
Also this helped on the server but not the nodes

sudo update-rc.d FOG*service* disable```

Try disabling the FOG Services from starting at boot.

Then, edit the /etc/rc.local file.

Add:

sleep 30
/etc/init.d/FOGPingHosts stop
/etc/init.d/FOGScheduler stop
/etc/init.d/FOGImageReplicator stop
/etc/init.d/FOGSnapinReplicator stop
/etc/init.d/FOGMulticastManager stop
/etc/init.d/FOGPingHosts start
/etc/init.d/FOGScheduler start
/etc/init.d/FOGImageReplicator start
/etc/init.d/FOGSnapinReplicator start
/etc/init.d/FOGMulticastManager start

boeleke · Mar 17, 2016, 9:34 AM

Guys , @Raymond-Bell & @Tom-Elliott

do you got an patch-solution for this?
After the patch from this afternoon i also get HIGH CPU services (now got 6777)

Server is freaking out
Ubuntu 14.04

Tom Elliott · Mar 17, 2016, 3:43 PM

@Raymond-Bell The rc.local stuff is also on the Storage Nodes?

Raymond Bell · Mar 17, 2016, 3:44 PM

@Tom-Elliott said:

@Raymond-Bell The rc.local stuff is also on the Storage Nodes?

yes i edited rc.local on Storage Nodes also

Tom Elliott · Mar 17, 2016, 3:45 PM

@Raymond-Bell Did you restart the services after adding the rc.local? I’m sorry if I’m asking obvious questions, but I really just need to make sure. Current update should help alleviate CPU usage as well though.

Raymond Bell · Mar 17, 2016, 3:46 PM

@Tom-Elliott said:

@Raymond-Bell Did you restart the services after adding the rc.local? I’m sorry if I’m asking obvious questions, but I really just need to make sure. Current update should help alleviate CPU usage as well though.

yes restarted and rebooted and also up to r5038 6777
will do it again to make sure

Raymond Bell · Mar 17, 2016, 3:49 PM

@Tom-Elliott Yes same result after

sudo service FOGMulticastManager stop
sudo service FOGImageReplicator stop
sudo service FOGSnapinReplicator stop
sudo service FOGScheduler stop
sudo service FOGPingHosts stop


sudo update-rc.d FOGMulticastManager disable
sudo update-rc.d FOGImageReplicator disable
sudo update-rc.d FOGSnapinReplicator disable
sudo update-rc.d FOGScheduler disable
sudo update-rc.d FOGPingHosts disable

sudo service FOGMulticastManager start
sudo service FOGImageReplicator start
sudo service FOGSnapinReplicator start
sudo service FOGScheduler start
sudo service FOGPingHosts start

top - 10:48:18 up  1:48,  2 users,  load average: 4.14, 4.92, 5.02
Tasks: 202 total,   6 running, 196 sleeping,   0 stopped,   0 zombie
%Cpu(s): 66.6 us, 32.7 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
KiB Mem:   1014980 total,   966204 used,    48776 free,    19852 buffers
KiB Swap:  1037308 total,    12316 used,  1024992 free.   529972 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 5970 root      20   0   41920  20164  15184 R  45.6  2.0   0:07.37 FOGSnapinReplic
 6026 root      20   0   41920  20392  15408 R  41.3  2.0   0:05.85 FOGPingHosts
 5990 root      20   0   41920  20240  15256 R  39.6  2.0   0:06.34 FOGTaskSchedule
 5950 root      20   0   41920  20488  15504 R  39.3  2.0   0:06.90 FOGImageReplica
 5930 root      20   0   41920  20116  15132 R  34.3  2.0   0:05.32 FOGMulticastMan
 5779 fog       20   0   11284   3900   3128 S   0.3  0.4   0:00.04 sshd
    1 root      20   0    4732   3788   2548 S   0.0  0.4   0:02.83 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.78 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   0:05.27 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.12 ksoftirqd/1

boeleke · Mar 17, 2016, 10:36 PM

Something weird. After the new update , my CPU performance from my server was going to 10-20%. After a reboot , he started again @100%

Pretty anoying

HIGH CPU Fog Services after update r5029 v6759

153

12.1k

17.3k

155.3k