FOG Server CPU usage 100%

Wayne Workman

@fernando-gietz said in FOG Server CPU usage 100%:

My hosts table has 7000 entries

Admittedly, this is a whole whole lot of hosts reporting to one server.

At a past job, we had in one setup maybe 2k hosts going to one fog server, that FOG Server has 8 cores, 8GB of RAM. It’s a pretty beefy server but it did a ton of work, and managed a ton of Technicians and a ton of images, and a ton of imaging and snapins and storage nodes. That box was Really busy, and that’s with 2K hosts… You say you have 7,000 hosts… I’d say you need a 16 core server, and configure MySQL to use more than one thread, use solid state disks to host the database, and assign 16GB of RAM - and never image from that server, make storage nodes handle the imaging loads - and keep those loads off of your main server. If you wanted to go even further, you could host the FOG Database on it’s own (very fast) server, and the rest of the main fog server on another box.

Fernando Gietz

After UEFI issue I have time to this XD

@wayne-workman said in FOG Server CPU usage 100%:

@fernando-gietz I was thinking about this - last time someone had this issue, it was the mysql max connections being hit.

Maybe one of the FOG updates or an OS update (especially likely with Ubuntu systems) wiped out our max connection setting.

Anyways, heres what might fix it:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_MySQL#Increase_maximum_simultaneous_MySQL_connections

Make sure to do the query in mysql AND to change the conf setting too so the change persists reboots!

I have increased the connections and the trouble follows
I have setup the mysql to log the slow queries and not appears nothing.

The problem is the apache and his activity, it reaches to 100% of CPU. If you see the capture of the browser activity, the problem begins when the browser tries to “Dibujar” (In english, Draw). I think that the browser must to draw all elements, visible and hidden ones.
If you see the membership page, you can see the list of the members, but the browser doesn’t show the members that are not in the group, to see them you need check the checkbox; then, appears all of them. This list is not create on-the-fly, the browser “drew” it before. In my case, the browser need to draw 7000 elements

Tom Elliott

@fernando-gietz The problem, as I’m understanding it, however, is on the Server, not the client machine. The “Drawing” should only be happening on the Browser, so the setTimeout is doing something a little funny, though I’m not 100% sure what.

Tom Elliott

Mind checking if the Location Plugin has a bunch of erroneous entries in it? I’ve seen/heard of people having a real hard time with CPU because the mysql database is just crammed full of data. The last time I saw it, there were a bunch (10’s of thousands) of entries in one of the tables. @Wayne-Workman May remember better than me. He sort of replicated this problem by accidentally uploading a binary data file (like teamviewer installer) into the CSV importer. This spammed the Database with TONS of data, and each request to the database would require a rescan on the HDD causing the CPU load being extremely high.

Sebastian Roth

Possibly it’s just that apache hits a wall with too many client requests and too little worker threads?

Tom Elliott

@sebastian-roth It’s possible, but less likely. While a ton of machines hitting the system at the same time can cause issues, I think the communication is so minuscule that I’m going to chalk this up to constant reloading of the Database, which is likely in a HUGE state.

@Fernando-Gietz You could try testing this theory by getting a backup of the current database, then clearing out the larger table sets, taskLog, historyLog, snapinJobs, imagingLog, etc…

Things that are “logging” or have no real value in being extremely large.

Wayne Workman

@tom-elliott said in FOG Server CPU usage 100%:

He sort of replicated this problem by accidentally uploading a binary data file (like teamviewer installer) into the CSV importer.

But you fixed that problem by verifying that the CSV uploaded was an actual CSV.

I would recommend cleaning out the DB though. Just truncate the history table, the tasks table, etc.

Tom Elliott

@wayne-workman I know what was fixed, but I also know WHY it was causing issues, it wasn’t the “bad” data persay, but rather the “amount” of bad data in the database.

Sifting through 5000 shreds of paper (let’s say a sheet of paper is 1000 shreds each) is much harder than sifting through 5 pieces of paper, right?

So size may not be the issue, rather the quantity held within the size.

Fernando Gietz

@wayne-workman The database size is not very big

+------------------------+---------------+---------------+
| Table Name             | Quant of Rows | Total Size Kb |
+------------------------+---------------+---------------+
| LDAPServers            |             1 |         14.32 |
| clientUpdates          |             0 |          4.00 |
| dirCleaner             |             0 |          1.00 |
| globalSettings         |           180 |         73.30 |
| greenFog               |             0 |          1.00 |
| groupMembers           |          6881 |        546.79 |
| groups                 |           313 |         53.92 |
| history                |          4268 |       1011.34 |
| hookEvents             |           277 |         31.68 |
| hostAutoLogOut         |           502 |         34.20 |
| hostMAC                |          7113 |        826.06 |
| hostScreenSettings     |           502 |         38.18 |
| hosts                  |          7008 |       1088.59 |
| imageGroupAssoc        |           279 |         18.03 |
| imagePartitionTypes    |            12 |          3.34 |
| imageTypes             |             4 |          3.22 |
| images                 |           279 |         72.12 |
| imagingLog             |          2537 |        273.77 |
| inventory              |           618 |        257.35 |
| ipxeTable              |          2097 |        171.71 |
| keySequence            |            35 |          2.79 |
| moduleStatusByHost     |          4932 |        403.68 |
| modules                |            13 |          5.66 |
| multicastSessions      |           181 |         26.43 |
| multicastSessionsAssoc |          2130 |        154.86 |
| nfsFailures            |             0 |          1.00 |
| nfsGroupMembers        |             1 |         20.22 |
| nfsGroups              |             1 |         13.05 |
| notifyEvents           |             5 |          9.14 |
| os                     |            12 |          3.26 |
+------------------------+---------------+---------------+

This server have 7000 entries in the hosts table but only is used by 400-500 computers

Fernando Gietz

In the develope environment I have other server, in this one I deleted 6500 entries and It works better.

+------------------------+---------------+---------------+
| Table Name             | Quant of Rows | Total Size Kb |
+------------------------+---------------+---------------+
| LDAPServers            |             1 |         14.32 |
| clientUpdates          |             0 |          1.00 |
| dirCleaner             |             0 |          1.00 |
| globalSettings         |           183 |         73.87 |
| greenFog               |             0 |          1.00 |
| groupMembers           |           279 |        534.84 |
| groups                 |            28 |         48.79 |
| history                |         25895 |       5894.44 |
| hookEvents             |           278 |         31.72 |
| hostAutoLogOut         |           108 |         12.11 |
| hostMAC                |          6938 |        787.91 |
| hostScreenSettings     |           109 |         13.51 |
| hosts                  |           480 |       1036.53 |
| imageGroupAssoc        |           292 |         18.15 |
| imagePartitionTypes    |            12 |          3.34 |
| imageTypes             |             4 |          3.22 |
| images                 |           292 |         71.45 |
| imagingLog             |          1735 |        165.28 |
| inventory              |           259 |         99.80 |
| ipxeTable              |          1620 |        123.98 |
| keySequence            |            35 |          2.79 |
| moduleStatusByHost     |          6097 |        489.08 |
| modules                |            13 |          5.66 |
| multicastSessions      |          1208 |        138.77 |
| multicastSessionsAssoc |         24929 |       1417.48 |
| nfsFailures            |             0 |          1.00 |
| nfsGroupMembers        |             1 |         12.18 |
| nfsGroups              |             1 |          7.05 |
| notifyEvents           |             5 |          9.14 |
| os                     |            12 |          3.26 |
+------------------------+---------------+---------------+

Wayne Workman

@fernando-gietz said in FOG Server CPU usage 100%:

The problem is the apache and his activity, it reaches to 100% of CPU.

Are you using image replication?

Fernando Gietz

@wayne-workman No. I have only one server and one node. But the service is running.

Wayne Workman

@fernando-gietz Turn it off please and see what happens. systemctl stop FOGImageReplicator or something like that.

Fernando Gietz

@wayne-workman Nothing. the behavour is the same. I have restarted the apache too.

Wayne Workman

@fernando-gietz Please read through this: https://stackoverflow.com/questions/32006559/show-which-php-scripts-apache-is-currently-running

We need to figure out what’s consuming all of your CPU. You say it’s httpd, but we don’t know anything further than that. Using the methods described in the thread above, you should be able to pin-point exactly which PHP script is consuming the CPU.

Thread from stackoverflow copy/pasted below:

You can get the current working directory and opened files using lsof. Sadly this does not include the script being run, but if scripts are in different directories, or open different files, you can distinguish between them. Eg the following shows a script in /var/www/stuff/php is running:

sudo lsof -c http | grep cwd
httpd   24475  id  cwd DIR 8,3     4096 1123403 /var/www/stuff/php

You can configure apache to let you have the information. Add an entry like

<Location /server-status>
    SetHandler server-status
    ...
</Location>

in your conf ensuring it is only made available to restricted remotes. Then browse to http://localhost/server-status and you will have a “ps” of running jobs. For example, this output shows gg.php is running (I’ve removed some columns):

Srv PID     Acc     M CPU     SS Slot Client  VHost  Request
0-0 23688   0/0/262 W 0.50    2  4.48 ::1     xxx:80 GET /php/gg.php HTTP/1.1
1-0 23743   0/0/187 W 0.00    0  3.30 ::1     xxx:80 GET /server-status HTTP/1.1

Wayne Workman

@fernando-gietz There is also the apachetop command that you might be able to use too. I almost forgot about this article but this shows how to use it: https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_Web_Interface Along with a list of historic threads where people have had high CPU problems in the past. (if we solve yours, it will be added to the list too).

Tom Elliott

@fernando-gietz I’m going to guess the tasks table is a bit on the high side maybe?

Fernando Gietz

Done.

[root@fog7 conf]# lsof -c http | grep cwd
httpd   13036   root  cwd       DIR              253,0      4096      128 /
httpd   13037 apache  cwd       DIR              253,2       133 23069131 /var/www/html/fog/management
httpd   13038 apache  cwd       DIR              253,0      4096      128 /
httpd   13039 apache  cwd       DIR              253,0      4096      128 /
httpd   13040 apache  cwd       DIR              253,0      4096      128 /
httpd   13041 apache  cwd       DIR              253,0      4096      128 /
httpd   13044 apache  cwd       DIR              253,2      4096  8388946 /var/www/html/fog/status
httpd   13118 apache  cwd       DIR              253,0      4096      128 /
httpd   13144 apache  cwd       DIR              253,0      4096      128 /
httpd   13145 apache  cwd       DIR              253,0      4096      128 /

And now the output of server status

Server Version: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.6.31
Server MPM: prefork
Server Built: Jul 26 2017 04:45:44

Current Time: Tuesday, 14-Nov-2017 19:49:35 CET
Restart Time: Tuesday, 14-Nov-2017 19:44:15 CET
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 5 minutes 20 seconds
Server load: 1.01 0.85 0.65
Total accesses: 557 - Total Traffic: 3.4 MB
CPU Usage: u167.78 s11.08 cu0 cs0 - 55.9% CPU load
1.74 requests/sec - 10.8 kB/second - 6.2 kB/request
3 requests currently being processed, 6 idle workers

W____WW__.......................................................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv	PID	Acc	M	CPU 	SS	Req	Conn	Child	Slot	Client	VHost	Request
0-0	13037	0/58/58	W 	5.22	120	0	0.0	0.19	0.19 	158.227.4.135	10.0.15.4:80	GET /fog/management/index.php?node=group&sub=membership&id=334
1-0	13038	0/75/75	_ 	6.88	2	107	0.0	0.27	0.27 	158.227.138.124	10.0.15.4:80	GET /fog/management/index.php?sub=requestClientInfo&configure&n
2-0	13039	0/76/76	_ 	6.59	2	0	0.0	0.23	0.23 	158.227.4.135	10.0.15.4:80	GET /server-status HTTP/1.1
3-0	13040	0/39/39	_ 	4.78	2	232	0.0	0.07	0.07 	158.227.138.124	10.0.15.4:80	GET /fog/management/index.php?sub=requestClientInfo&mac=94:57:A
4-0	13041	0/75/75	_ 	6.74	1	150	0.0	0.23	0.23 	158.227.115.42	10.0.15.4:80	GET /fog/management/index.php?node=task&sub=active&_=1510657597
5-0	13044	0/19/19	W 	131.09	87	0	0.0	1.69	1.69 	158.227.4.135	10.0.15.4:80	POST /fog/status/getservertime.php HTTP/1.1
6-0	13118	0/72/72	W 	7.25	0	0	0.0	0.23	0.23 	158.227.4.135	10.0.15.4:80	GET /server-status HTTP/1.1
7-0	13144	0/72/72	_ 	6.11	1	73	0.0	0.23	0.23 	158.227.138.124	10.0.15.4:80	GET /fog/service/getversion.php?newService&json HTTP/1.1
8-0	13145	0/71/71	_ 	4.20	2	75	0.0	0.24	0.24 	158.227.138.124	10.0.15.4:80	GET /fog/service/getversion.php?clientver&newService&json HTTP/
Srv	Child Server number - generation
PID	OS process ID
Acc	Number of accesses this connection / this child / this slot
M	Mode of operation
CPU	CPU usage, number of seconds
SS	Seconds since beginning of most recent request
Req	Milliseconds required to process most recent request
Conn	Kilobytes transferred this connection
Child	Megabytes transferred this child
Slot	Total megabytes transferred this slot
SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current entries: 0
subcaches: 32, indexes per subcache: 88
index usage: 0%, cache usage: 0%
total entries stored since starting: 0
total entries replaced since starting: 0
total entries expired since starting: 0
total (pre-expiry) entries scrolled out of the cache: 0
total retrieves since starting: 0 hit, 0 miss
total removes since starting: 0 hit, 0 miss

Fernando Gietz

@fernando-gietz

/fog/management/index.php?node=group&sub=membership&id=334

Wayne Workman

Actually this line is using a ton of CPU:

131.09	87	0	0.0	1.69	1.69 	158.227.4.135	10.0.15.4:80	POST /fog/status/getservertime.php HTTP/1.1

131 is a LOT.

FOG Server CPU usage 100%

144

12.0k

17.3k

155.1k