More info about this 🙂
Now we have configurated the client cheout time = 275 seconds. In September we had to increase it to 900, but after the changes we have setup it as I tell, 275 seconds.
This is a capture of goaccess tool:
0_1540379551079_goaccess_RHEL.png
The capture is after 6 minutes of activity, we can see that:
Total request: 8436 -> 23,5 req/sec
1378 visitors requested the /fog/service/getversion.php?NewServise&json page. This says that 1378 client are connected simultaneous.
1 visitor requested /fog/service/progress.php 114 times. This client is doing a download task.
Is necessary take in account that when a cleint is doing a dowload, capture or multicast task, this client asks or reports to the server his advance (yes is very pretty and cool see the progress bar) but this info has a price. The client reports his advance more or lees every three or four seconds, when you have one tasks is “pecata minuta”, but when you have 100 or more client doing download or cpature tasks is a problem, because the php server can not process all request simultaneous and is not only the php server, is the mysql server too.
For example in this capture of htop command (like atop or top):
0_1540380690148_Captura de pantalla de 2018-10-24 13-08-54.png
We can see that the vCores are busy, but not at 100%, the load is high, mysqld is using 221% of CPU. In this moment the server is proccessing only the FOG client requests of the computers, there is no any tasks (When the technicians send download, multicast or capture tasks, the server is burning … literally. I saw the load at 60 or more, the server could not attend the all request and refused them ), In this capture shows the activity of the two NUMA nodes clearly.
Node 0: 1, 2 ,3 and 4
Node 1: 5 ,6, 7 and 8
Where is working the mysqld proccess?
# ./numa-maps-summary.pl < /proc/1787/numa_maps
N0 : 636053 ( 2.43 GB)
N1 : 14336 ( 0.05 GB)
active : 352378 ( 1.34 GB)
anon : 649153 ( 2.48 GB)
dirty : 649153 ( 2.48 GB)
kernelpagesize_kB: 1016 ( 0.00 GB)
mapmax : 480 ( 0.00 GB)
mapped : 1276 ( 0.00 GB)
In the Node 0.
I downloaded a little script, i forgot from where, that shows the usage of RAM of each proccess:
# ./ps_mem.py
Private + Shared = RAM used Program
4.0 KiB + 12.5 KiB = 16.5 KiB agetty
4.0 KiB + 15.0 KiB = 19.0 KiB mysqld_safe
4.0 KiB + 47.5 KiB = 51.5 KiB rpc.statd
4.0 KiB + 49.5 KiB = 53.5 KiB rpc.idmapd
4.0 KiB + 57.0 KiB = 61.0 KiB lvmetad
36.0 KiB + 31.0 KiB = 67.0 KiB atd
4.0 KiB + 73.5 KiB = 77.5 KiB VGAuthService
88.0 KiB + 32.0 KiB = 120.0 KiB rhsmcertd
92.0 KiB + 41.5 KiB = 133.5 KiB systemd-udevd
112.0 KiB + 22.5 KiB = 134.5 KiB sleep
88.0 KiB + 55.0 KiB = 143.0 KiB vsftpd
88.0 KiB + 65.0 KiB = 153.0 KiB gssproxy
148.0 KiB + 23.0 KiB = 171.0 KiB udp-sender
156.0 KiB + 30.0 KiB = 186.0 KiB crond
164.0 KiB + 30.0 KiB = 194.0 KiB in.tftpd
180.0 KiB + 20.0 KiB = 200.0 KiB numad
192.0 KiB + 16.0 KiB = 208.0 KiB rhnsd
128.0 KiB + 83.5 KiB = 211.5 KiB master
176.0 KiB + 35.5 KiB = 211.5 KiB xinetd
188.0 KiB + 54.5 KiB = 242.5 KiB auditd
232.0 KiB + 29.0 KiB = 261.0 KiB irqbalance
192.0 KiB + 88.5 KiB = 280.5 KiB qmgr
208.0 KiB + 87.0 KiB = 295.0 KiB sh
240.0 KiB + 109.5 KiB = 349.5 KiB rpcbind
588.0 KiB + 36.5 KiB = 624.5 KiB systemd-logind
668.0 KiB + 75.5 KiB = 743.5 KiB dbus-daemon
596.0 KiB + 311.5 KiB = 907.5 KiB polkitd
800.0 KiB + 175.5 KiB = 975.5 KiB vmtoolsd
916.0 KiB + 64.5 KiB = 980.5 KiB FOGpxe.sh
936.0 KiB + 135.0 KiB = 1.0 MiB dnsmasq
1.1 MiB + 386.0 KiB = 1.5 MiB NetworkManager
1.4 MiB + 413.5 KiB = 1.8 MiB pickup
2.0 MiB + 101.0 KiB = 2.1 MiB rpc.mountd
2.2 MiB + 67.5 KiB = 2.3 MiB systemd
2.6 MiB + 308.0 KiB = 2.9 MiB tuned
2.9 MiB + 355.5 KiB = 3.2 MiB mysql
2.7 MiB + 846.0 KiB = 3.5 MiB bash (6)
3.0 MiB + 822.5 KiB = 3.8 MiB FOGSnapinReplic (2)
3.1 MiB + 682.5 KiB = 3.8 MiB FOGImageReplica (2)
4.2 MiB + 80.0 KiB = 4.2 MiB nsrexecd
4.1 MiB + 718.5 KiB = 4.8 MiB FOGSnapinHash (2)
3.6 MiB + 1.3 MiB = 4.9 MiB sudo (3)
5.4 MiB + 1.0 MiB = 6.4 MiB FOGTaskSchedule (2)
7.5 MiB + 118.5 KiB = 7.6 MiB glusterfsd
1.9 MiB + 6.1 MiB = 8.0 MiB sshd (7)
2.5 MiB + 7.1 MiB = 9.5 MiB rsyslogd
10.3 MiB + 713.5 KiB = 11.0 MiB FOGImageSize (2)
7.0 MiB + 9.9 MiB = 16.9 MiB systemd-journald
21.5 MiB + 791.0 KiB = 22.2 MiB FOGPingHosts (2)
25.6 MiB + 1.0 MiB = 26.6 MiB FOGMulticastMan (2)
315.4 MiB + 14.6 MiB = 330.0 MiB php-fpm (51)
2.5 GiB + 289.0 KiB = 2.5 GiB mysqld
5.1 GiB + 14.9 MiB = 5.1 GiB httpd (12)
---------------------------------
8.0 GiB
=================================
