CPU Usage high

[Taspharel]

Re: Support regarding CPU usage of FOG server

Hello again.

So, the CPU Usage is still very high on our FOG Server running in our Virtual Machine. I get an error about high CPU Usage in the vcenter.

Looking at the ‘top’ command I get:

PID    |   USER  |  PR |  NI |  VIRT   |       RES     |   SHR  |  S  |  %CPU |   %MEM |   Time+  |  COMMAND
12319 | apache  | 20   | 0  |  678888  |   25616  |  8032 | R  |   99.7    |   0.3       |   30414:05  |  httpd
15481| root     |   20 |   0  |  344072  |   13316  |  3220 |  R  |   99.7    |   0.2     |  30414:31 |  FOGTaskSchedule
23216 | apache | 20 |   0  |  521388   |  22732  |  5728 |  R  |  99.7   |    0.3    | 175:46.18 | httpd  

as the top three processes, everything else doesnt seem to tax the cpu at all (0.0 or 0.1 mostly for CPU%)

The VM is set up with 4 CPUs with 2 Cores. There is about 20 servers running on that VM Server, none other seem to have CPU issues.

We have about 160 Machines connected to the FOG server at very peak times, usually about 60-80, no Tasks scheduled (due to it still not working for us, but thats a different story) and no replication set up.
There is about 6 images on the server (including Hirens Boot CD and two other “tool menu options”), 2 of which we use now and then to update single machines to the newest golden win10 image version.

The only person accessing the fog server to manage it is me from one computer only. Closing the browser windows on my client that access the FOG UI doesnt seem to change anything in the ‘top’ command.

Running:
Running Version 1.5.0-RC-10
SVN Revision: 6080

On CentOS, up to date as of a few days ago, problem has persisted for longer than that.

Would any other information help us in solving this? Thanks for your support!

Update: I ran the database cleaning commands from the wiki - no change.

And the top of the ‘top’ command shows:

top - 14:27:09 up 72 days, 20:01,  1 user,  load average: 3.01, 3.38, 3.28
Tasks: 198 total,   4 running, 194 sleeping,   0 stopped,   0 zombie
%Cpu(s): 75.2 us,  0.2 sy,  0.0 ni, 24.5 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  8010576 total,  1167556 free,   848008 used,  5995012 buff/cache
KiB Swap:  4063228 total,  4063228 free,        0 used.  6760720 avail Mem

[Sebastian Roth]

@Taspharel This is even after rebooting the FOG server? Still FOGTaskSchedule with high CPU? Do you see anything useful in the apache logs (see my signature on where to find those)? As well, post the contents of your /var/log/fog/fogscheduler.log here.

[Quazz]

@taspharel said in CPU Usage high:

no Tasks scheduled (due to it still not working for us, but thats a different story)

Seems rather relevant to me if FOGTaskSchedule is taking up so much CPU while not working.

What happens when you try to schedule a task?

When you restart apache, how long does it function normally before CPU goes to 100?

[Wayne Workman]

@sebastian-roth said in CPU Usage high:

As well, post the contents of your /var/log/fog/fogscheduler.log here.

That log will be most important in this situation. Have logs moved? I remember them being inside /opt/fog/log ?

[Sebastian Roth]

@Wayne-Workman This is a link… /opt/fog/log is actually the place of the logs. As most people are used to /var/log I mostly use this.

[Taspharel]

Fogscheduler Log:

[12-13-17 11:13:05 am]
==================================
===        ====    =====      ====
===  =========  ==  ===   ==   ===
===  ========  ====  ==  ====  ===
===  ========  ====  ==  =========
===      ====  ====  ==  =========
===  ========  ====  ==  ===   ===
===  ========  ====  ==  ====  ===
===  =========  ==  ===   ==   ===
===  ==========    =====      ====
==================================
===== Free Opensource Ghost ======
==================================
============ Credits =============
= https://fogproject.org/Credits =
==================================
== Released under GPL Version 3 ==
==================================


[12-13-17 11:13:05 am] Interface Ready with IP Address: 127.0.0.1
[12-13-17 11:13:05 am] Interface Ready with IP Address: 127.0.1.1
[12-13-17 11:13:05 am] Interface Ready with IP Address: 192.168.43.17
[12-13-17 11:13:05 am] Interface Ready with IP Address: fogserver
[12-13-17 11:13:05 am]  * Starting TaskScheduler Service
[12-13-17 11:13:05 am]  * Checking for new items every 60 seconds
[12-13-17 11:13:05 am]  * Starting service loop
[12-13-17 11:13:05 am]  * 3 tasks found.


**And the Apache error log:**

[Mon Jan 01 03:38:22.208005 2018] [auth_digest:notice] [pid 12312] AH01757: generating secret for digest authentication ...
[Mon Jan 01 03:38:22.208797 2018] [lbmethod_heartbeat:notice] [pid 12312] AH02282: No slotmem from mod_heartmonitor
[Mon Jan 01 03:38:22.225832 2018] [mpm_prefork:notice] [pid 12312] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.6.32 configured -- resuming normal operations
[Mon Jan 01 03:38:22.225842 2018] [core:notice] [pid 12312] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Jan 03 11:18:17.743911 2018] [:error] [pid 10603] [client 192.168.43.93:25534] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 35505 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=165
[Wed Jan 03 11:18:48.656285 2018] [:error] [pid 23214] [client 192.168.43.93:25548] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 35505 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=165
[Wed Jan 03 11:20:26.383096 2018] [:error] [pid 23214] [client 192.168.43.93:25585] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 35505 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=165
[Wed Jan 03 11:26:43.136156 2018] [:error] [pid 23402] [client 192.168.43.93:25747] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 35505 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=165
[Wed Jan 03 11:29:43.869919 2018] [:error] [pid 10603] [client 192.168.43.93:25822] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 35505 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=165
[Wed Jan 03 11:48:19.361457 2018] [:error] [pid 10603] [client 192.168.43.93:26266] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 35505 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=165
[Wed Jan 03 11:54:39.077064 2018] [:error] [pid 23270] [client 192.168.43.93:26420] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 35505 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=165
[Wed Jan 03 11:56:01.568297 2018] [:error] [pid 23305] [client 192.168.43.93:26450] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 35505 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=165
[Wed Jan 03 12:20:25.820408 2018] [:error] [pid 23305] [client 192.168.43.93:26935] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 16384 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=165
[Wed Jan 03 12:37:30.868478 2018] [:error] [pid 23305] [client 192.168.43.93:27362] PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 49152 bytes) in /var/www/html/fog/lib/fog/fogcontroller.class.php on line 261, referer: http://fog.lfdw.local/fog/management/index.php?node=host&sub=edit&id=127

Regarding the schedules:
I still have the problem that when trying to show my scheduled tasks the web gui hangs and I cannot open any more pages on the webgui until I delete my browsers data (cookies, etc.) - tried with several different browsers and seems (at least according to what i took from the thread where I posted that) to be a problem with my setup only … since we dont necessarily need the scheduled tasks I just ignored it so far.

Will try a reboot now.

Update:
Rebooting the machine puts the FOGTaskSchedule at the top with 100 % CPU again, 0.2 % Mem … the httpd ones are still at the top but with much lower cpu values, sometimes going down to 0.

[Sebastian Roth]

@Taspharel Please run the following database queries to see what tasks are still active.

shell> mysql -u root -p
Enter password:
...
mysql> use fog;
...
mysql> SELECT * FROM scheduledTasks WHERE stActive=1;
...
mysql> SELECT * FROM powerManagement WHERE pmAction LIKE '%wol%';
...
mysql> quit;
shell>

[Taspharel]

First command results in:

Second one:

[Sebastian Roth]

@Taspharel As you said you don’t need those tasks I’d either delete those from the database or simply unset the active bit:

either disable:

shell> mysql -u root -p
Enter password:
...
mysql> use fog;
...
mysql> UPDATE scheduledTasks SET stActive=0 WHERE stActive=1;
...
mysql> quit;
shell>

or delete:

shell> mysql -u root -p
Enter password:
...
mysql> use fog;
...
mysql> DELETE FROM scheduledTasks WHERE stActive=1;
...
mysql> quit;
shell>

Around 60 seconds later the CPU load should drop. If not you can try restarting the server.

[Taspharel]

Thanks for the help, disabled the active bit and after a reboot ‘top’ no longer shows any process with high CPU usage in idle state.

In the long term the question remains: Why did that happen at all and is it related to our error regarding the web-gui “freeze” when looking at scheduled tasks?

Update:
Just checked: I can now open the “Scheduled Tasks” website in the FOG-Gui without problems …

[Sebastian Roth]

@taspharel said in CPU Usage high:

In the long term the question remains: Why did that happen at all and is it related to our error regarding the web-gui “freeze” when looking at scheduled tasks?

Well it might be related as in the “freeze” might have been caused by the same “zombie” jobs. But I don’t really know.

Looking at the three tasks in the database again I see that all of them had stImageID zero. Don’t think that is valid. Not sure if that was causing it.