Very high CPU usage httpd, mysqld, FOGMulticastManager FOG trunk@5224
-
I am wondering if this is the same issue we seem to see in environments with a lot of clients lately…?? @Tom-Elliott any news on this issue? Unfortunately I wasn’t able to follow this as my new job just started this week.
Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)
Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG
-
@Sebastian-Roth I haven’t got a clue or any status. I’ve added a bit more checking and it seems extremely random as to what causes the issue(s). I only know of one case where it’s constantly plaguing ( @Raymond-Bell ) And the others seem to do better after resetting all encryption data to all hosts.
-
@Tom-Elliott Morning Tom
I upgraded late yesterday afternoon 26Apr GMT to the latest git release:
git-svn-id: https://svn.code.sf.net/p/freeghost/code/trunk@5303 71f96598-fa45-0410-b640-bcd6f8691b32
I’m still seeing high CPU usage. top shows CPU jumping between 25-65%
top - 11:20:20 up 6 days, 19:45, 1 user, load average: 5.31, 4.77, 4.80 Tasks: 154 total, 3 running, 151 sleeping, 0 stopped, 0 zombie Cpu(s): 50.4%us, 25.3%sy, 0.0%ni, 23.8%id, 0.0%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 1922092k total, 1720328k used, 201764k free, 86980k buffers Swap: 4128764k total, 5512k used, 4123252k free, 1253364k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6726 mysql 20 0 1350m 51m 6596 S 11.3 2.8 103:15.15 mysqld 30973 apache 20 0 327m 13m 3856 S 9.3 0.7 1:30.69 httpd 31311 apache 20 0 327m 13m 3856 S 9.3 0.7 0:19.28 httpd 30135 apache 20 0 328m 13m 3856 R 9.0 0.7 4:06.95 httpd 30754 apache 20 0 328m 13m 3856 S 9.0 0.7 2:12.27 httpd 31364 apache 20 0 327m 13m 3856 S 9.0 0.7 0:10.67 httpd 30861 apache 20 0 327m 13m 3856 S 8.6 0.7 1:54.65 httpd 31356 apache 20 0 328m 13m 3856 S 8.6 0.7 0:11.72 httpd 31412 apache 20 0 327m 13m 3856 R 8.3 0.7 0:01.23 httpd 30728 apache 20 0 328m 13m 3856 S 7.6 0.7 2:13.29 httpd 31361 apache 20 0 328m 13m 3856 S 7.3 0.7 0:10.60 httpd 31403 apache 20 0 327m 13m 3716 S 6.6 0.7 0:02.80 httpd 30744 apache 20 0 327m 13m 3856 S 6.3 0.7 2:12.84 httpd 31314 apache 20 0 327m 13m 3856 S 6.0 0.7 0:19.26 httpd 31363 apache 20 0 328m 13m 3856 S 6.0 0.7 0:10.76 httpd 31400 apache 20 0 328m 13m 3856 S 6.0 0.7 0:02.77 httpd 31406 apache 20 0 327m 13m 3856 S 6.0 0.7 0:02.67 httpd 31414 apache 20 0 327m 13m 3848 S 6.0 0.7 0:01.41 httpd 31006 apache 20 0 328m 13m 3856 S 4.7 0.7 1:23.89 httpd 30857 apache 20 0 327m 13m 3856 S 3.7 0.7 1:53.48 httpd 30751 apache 20 0 327m 13m 3856 S 3.3 0.7 2:11.88 httpd 9083 root 20 0 318m 24m 3528 S 1.7 1.3 2:59.69 FOGPingHosts 11 root 20 0 0 0 0 S 0.3 0.0 4:23.42 events/0This is happening on 2 servers, FOG01 with 267 hosts, and FOG02 which has 124 hosts. FOG02 has high-ish CPU usage also but not anywhere near as high as FOG01. Both servers are identical in terms of OS and FOG versions, located in same site albeit FOG01 is serving local clients and FOG02 our remote sites. FOG02 will eventually have the WOL and location plugins installed for this resason but does not currently.
Ref removing the old client in case that’s interfering can we use any fogservice.msi installer or must it be the exact same as was installed on the client?
I am also seeing apache errors in log as follows, multiple IP addresses listed with same error
[Wed Apr 27 12:07:57 2016] [error] [client 10.166.***.***] PHP Warning: implode(): Invalid arguments passed in /var/www/html/fog/lib/fog/eventmanager.class.php on line 67Line in question in file refers to this one
$pluginfiles = array_values(array_filter(preg_grep(sprintf('#/(%s)/#',implode('|',$_SESSION['PluginsInstalled'])),array_map($fileitems,(array)$files))));I had plugins installed on this box previously but now do not so that could be all that means…
regards Kiweegie.
-
@Kiweegie Mind updating again?
-
@Tom-Elliott hi Tom, updated once more and still the same CPU usage - FOG01 around mid 60% max and FOG02 around mid 30% max.
git-svn-id: https://svn.code.sf.net/p/freeghost/code/trunk@5312 71f96598-fa45-0410-b640-bcd6f8691b32
Added to that I cannot access the Storage management, Report management or FOG configuration page links within the web GUI any longer… All other links are ok.
cheers Kiweegie.
-
@Kiweegie can you install
apachetopon your fog servers, and then run it? To run it after installed, just typeapachetopinstall on debian/ubuntu:
sudo apt-get install apachetop -yinstall on fedora/centOS/rhel:
yum install apachetop -yLet the app run for a while, maybe 15 or so minutes, and then give us a screenshot of what it’s showing. What’s important is the request count and data sent. We will be able to see what web file is getting the most activity and it’ll help us to narrow things down.
-
@Wayne-Workman Not a problem. Installed on both boxes and currently running - I need to run myself shortly but will fire over the results before i go in about 20 mins.
cheers, Kiweegie.
-
Some info for people experiencing high loads:
v0.10.0 of the client (currently undergoing release candidate testing) drastically cuts the amount of traffic used. In the “heaviest” cycle, the client will only make 3 requests. 1 to authenticate, 1 to get module settings, and 1 to get server settings (cycle time, client version,…). It also prevents run-away authentication, where the client had the potential to constantly spam the server’s authentication method.
Hopefully when v0.10.0 is released it should resolve any load issues. However, more than likely v0.10.0 may contain several bugs as the entire code base has changed to support any OS, and a completely new middleware API has been made to support the decrease in traffic. Since so much has changed, its almost guaranteed that I will have overlooked some minor things.
-
@Wayne-Workman Hi Wayne
here’s the output of apachetop on my main machine
last hit: 15:16:16 atop runtime: 0 days, 00:21:30 15:16:17 All: 22036 reqs ( 17.1/sec) 4721.2K ( 3747.7B/sec) 219.4B/req 2xx: 22036 ( 100%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%) R ( 30s): 454 reqs ( 15.1/sec) 90.0K ( 3072.1B/sec) 203.0B/req 2xx: 454 ( 100%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%) REQS REQ/S KB KB/S URL 210 7.00 1.0 0.0*/fog/service/servicemodule-active.php 53 1.77 86.9 2.9 /fog/management/other/ssl/srvpublic.crt 43 1.43 0.5 0.0 /fog/service/Printers.php 30 1.00 0.2 0.0 /fog/service/autologout.php 28 0.97 0.1 0.0 /fog/service/snapins.checkin.php 17 0.59 0.9 0.0 /fog/management/index.php 16 0.53 0.1 0.0 /fog/service/greenfog.php 15 0.50 0.1 0.0 /fog/service/printerlisting.php 14 0.47 0.1 0.0 /fog/service/getversion.php 14 0.48 0.1 0.0 /fog/service/jobs.php 13 0.45 0.1 0.0 /fog/service/hostname.php 1 0.04 0.0 0.0 *and from my second machine
last hit: 15:15:47 atop runtime: 0 days, 00:21:10 15:15:48 All: 5274 reqs ( 4.2/sec) 1008.8K ( 813.4B/sec) 195.9B/req 2xx: 5259 (99.7%) 3xx: 15 ( 0.3%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%) R ( 30s): 190 reqs ( 6.3/sec) 50.5K ( 1722.4B/sec) 272.0B/req 2xx: 190 ( 100%) 3xx: 0 ( 0.0%) 4xx: 0 ( 0.0%) 5xx: 0 ( 0.0%) REQS REQ/S KB KB/S URL 80 2.67 0.4 0.0*/fog/service/servicemodule-active.php 30 1.07 49.2 1.8 /fog/management/other/ssl/srvpublic.crt 13 0.45 0.2 0.0 /fog/service/Printers.php 9 0.35 0.0 0.0 /fog/service/hostname.php 9 0.33 0.1 0.0 /fog/service/getversion.php 9 0.35 0.0 0.0 /fog/service/jobs.php 9 0.38 0.0 0.0 /fog/service/printerlisting.php 9 0.36 0.0 0.0 /fog/service/snapins.checkin.php 9 0.38 0.0 0.0 /fog/service/greenfog.php 8 0.35 0.4 0.0 /fog/management/index.php 5 0.17 0.0 0.0 /fog/service/autologout.phpHeading off home now but will be online later if more information required.
regards Kiweegie.
-
Ok, @Developers what does
servicemodule-active.phpdo? On Kiweegie’s servers, it’s getting 7 requests per second. -
For each module the client has, it does 1 request to servicemodule-active to see if its enabled. That means, every 1 cycle of the client it’s calling that file about 5-6 times. This is one of the things the v0.10.0 removes completely.
Keep in mind the new client was originally designed for a completely different form of server<-> client communication (socket connections). And inorder to be able to release it in a timely fashion, we retrofitted the legacy client’s communication method onto the new client. This is why we have so much “useless” network traffic. v0.10.0 provides a hybrid of sorts between the ideal network communication model we originally wanted, and the legacy one.
-
Just pushed a fix to hopefully make things functional again (meaning cancelling/deleting/viewing) everything
-
@Tom-Elliott Hey Tom, many thanks. Both servers now have access to the reports, fog settings etc pages. CPU usage still high but the servers are usable enough albeit slow. I’ll keep an eye on Jbobs progress with the new client.
thanks again, Kiweegie.
-
@Tom-Elliott Hey Tom
another anomaly turned up since most recent changes. My Helpdesk team complained that all desktops with FOG client installed were boot looping. Work around was to remove all clients from the host management section in FOG. This permitted desktops to boot and not restart - suspect this is down to the old FOG client.
Only today getting time investigate this and noticed that when adding in a brand new host it seems to accept ok but nothing shows in the GUI when clicking List all hosts. If I export the list it shows one host only (I’ve added 2) but if I try and re-add the 2nd host i get error Hostname already exists.
Something funky going on with the database?
cheers Kiweegie
-
@Kiweegie go into mysql via CLI and look at the actual data, and let us see what you see.
mysql
use fog
select * from hosts; -
@Kiweegie Lots of improvements in the last week to the new client, can you update and see if this affects the CPU load at all?
I suspect there will be a quite dramatic decrease after moving to client 010.5
-
@Wayne-Workman Hi Wayne
apologies for not getting back to you til now, other projects keep piling up… I’ve checked the DB as you suggested via the cli and although only 3 hosts are listed in the GUI the cli shows 270 hosts…
| 270 | FOG01-current | | | 3 | 0 | 2016-05-21 11:28:20 | 2016-05-21 11:44:32 | foguser | 0 | | | | | | | | | | | | | | 0000-00-00 00:00:00 | 6 | | | 1 |Above is last snippet.
I guess I can just drop the DB and start over?
cheers Kiweegie.
-
@Kiweegie There have been a lot of improvements in the last two weeks! Mainly in the FOG client. Please upgrade to the latest version (first try the client, then FOG server) and see if things have changed for you as well.
-
@Sebastian-Roth Just upgraded trunk (5554) using git now and getting this error on install
./bin/installfog.sh: line 452: installPackages: command not found * Confirming package installation ./bin/installfog.sh: line 456: confirmPackageInstallation: command not found * Configuring services * What is the storage location for your images directory? (/images) -
@Kiweegie You have to run installfog.sh while inside the bin directory otherwise it will fail in more way than one (just checked this on my installation to be sure)
