FOG Multicast Not starting anymore
-
@lebrun78 You might ask, why did I have you change the engine and how will that help with performance?
The simple answer is MyISAM uses table level locking when something is updated. INNODB engine uses row level locking on an update. Under table level locking when someone goes to update a table, that processing updating the a row causes all other processes to stop updating until the write is complete. Under row level locking, only the database row that is being changed is locked all other tables and rows are free to be updated by other processes. The table level locking of MyISAM will create a back log of processes waiting their turn to update the database.
You will see this backlog by higher than normal cpu percent on the mysql process and many (many) php-fpm processes waiting their turn to update the database.
-
Il have allways too many time wait on port 9000
[root@fogus ~]# netstat -nat |grep 9000 |wc -l 855
and php-fpm is allways high
top - 17:08:40 up 6 days, 3:39, 2 users, load average: 0,69, 0,65, 0,79 Tasks: 339 total, 2 running, 337 sleeping, 0 stopped, 0 zombie %Cpu(s): 6,5 us, 2,6 sy, 0,0 ni, 90,8 id, 0,0 wa, 0,1 hi, 0,1 si, 0,0 st MiB Mem : 15328,7 total, 571,3 free, 1274,2 used, 13483,2 buff/cache MiB Swap: 7812,0 total, 7138,7 free, 673,2 used. 13484,4 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8776 mysql 20 0 3256016 464360 35664 S 17,9 3,0 2:31.14 mysqld 11704 apache 20 0 257736 21844 13380 S 14,6 0,1 0:04.37 php-fpm 11375 apache 20 0 257812 21868 13372 S 13,3 0,1 0:05.11 php-fpm 11608 apache 20 0 257816 21868 13376 S 13,0 0,1 0:05.32 php-fpm 11737 apache 20 0 257724 21840 13380 R 12,6 0,1 0:03.22 php-fpm 11376 apache 20 0 257736 21840 13372 S 12,0 0,1 0:05.33 php-fpm 11374 apache 20 0 257716 21276 12860 S 11,6 0,1 0:05.09 php-fpm 11377 apache 20 0 254816 21496 13052 S 11,6 0,1 0:05.17 php-fpm 11772 apache 20 0 254816 21436 12992 S 11,6 0,1 0:02.24 php-fpm 11373 apache 20 0 257724 21900 13372 S 11,3 0,1 0:05.16 php-fpm 11396 apache 20 0 2599116 17408 8900 S 5,0 0,1 0:01.30 httpd 11620 apache 20 0 2599116 17840 8948 S 5,0 0,1 0:02.10 httpd 11395 apache 20 0 2599116 19356 8920 S 1,7 0,1 0:00.63 httpd 11394 apache 20 0 2795788 23072 8948 S 0,7 0,1 0:00.33 httpd 1 root 20 0 244492 12360 7816 S 0,3 0,1 19:50.18 systemd 1173 root 20 0 200068 98172 23916 S 0,3 0,6 7:26.78 systemd-journal 3269 root 20 0 289304 16636 9884 S 0,3 0,1 7:34.19 php 11823 root 20 0 64176 5080 4152 R 0,3 0,0 0:00.03 top
-
@lebrun78 Well that didn’t seem to fix the issue. It should have, but you still have a long waiting list. How many client computers do you have in your network where the fog client is installed?
-
@lebrun78 looking at
top
screen it looks like you are using quite a bit of RAM and even dipping into swap space. Did you increase the number of php-fpm workers or increase the per worker memory limit at one time. These changes would have been made to the php-fpm config file. -
@lebrun78 I can imagine the php-fpm processes being under some heavy load some time after things were kind of offline because many clients try to check in again in a short time. But if things are fine this should “cool down” after an hour (even earlier but surely after that much time).
So if it’s still high load in php-fpm processes I suggest you also look at the Apache logs (see my signatur) and maybe even fog-client logs on machines (C:\fog.log). Post log information here so we can help interpret it.
-
@sebastian-roth
Hello Sebastian,
Here is the tail of /var/log/httpd/error.log file:[Fri Jan 14 08:46:48.363434 2022] [ssl:info] [pid 11396:tid 140375024985856] [client ip.lan.3.77:25158] AH01964: Connection to child 134 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:48.706587 2022] [ssl:info] [pid 11620:tid 140375417497344] [client ip.lan.6.71:59922] AH01964: Connection to child 192 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:48.768993 2022] [ssl:info] [pid 11620:tid 140375400711936] [client ip.lan.12.61:61280] AH01964: Connection to child 194 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:48.858436 2022] [ssl:info] [pid 11620:tid 140375134025472] [client ip.lan.12.61:61281] AH01964: Connection to child 201 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:48.868198 2022] [ssl:info] [pid 11620:tid 140374957876992] [client ip.lan.6.71:59924] AH01964: Connection to child 205 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:48.961868 2022] [ssl:info] [pid 11620:tid 140374597154560] [client ip.lan.12.61:61282] AH01964: Connection to child 215 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:48.990680 2022] [ssl:info] [pid 11620:tid 140375167596288] [client ip.lan.3.11:9250] AH01964: Connection to child 197 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:49.011505 2022] [ssl:info] [pid 11620:tid 140375383926528] [client ip.lan.12.61:61283] AH01964: Connection to child 196 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:49.088755 2022] [ssl:info] [pid 11620:tid 140375409104640] [client ip.lan.6.71:59926] AH01964: Connection to child 193 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:49.093387 2022] [ssl:info] [pid 11620:tid 140375117240064] [client ip.lan.3.11:9251] AH01964: Connection to child 203 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:49.197262 2022] [ssl:info] [pid 11620:tid 140374588761856] [client ip.lan.3.11:9252] AH01964: Connection to child 216 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:49.249588 2022] [ssl:info] [pid 11620:tid 140375142418176] [client ip.lan.3.11:9253] AH01964: Connection to child 200 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:49.264227 2022] [ssl:info] [pid 11620:tid 140374932698880] [client ip.lan.6.71:59928] AH01964: Connection to child 208 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:49.843373 2022] [ssl:info] [pid 11395:tid 140374446151424] [client ip.lan.7.63:57699] AH01964: Connection to child 82 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:49.966565 2022] [ssl:info] [pid 11395:tid 140375033378560] [client ip.lan.7.63:57700] AH01964: Connection to child 69 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.098427 2022] [ssl:info] [pid 11620:tid 140374622332672] [client ip.lan.7.63:57701] AH01964: Connection to child 212 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.103237 2022] [ssl:info] [pid 11396:tid 140375033378560] [client ip.lan.12.42:65003] AH01964: Connection to child 133 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.163407 2022] [ssl:info] [pid 11396:tid 140375417497344] [client ip.lan.7.63:57702] AH01964: Connection to child 128 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.181040 2022] [ssl:info] [pid 11620:tid 140375150810880] [client ip.lan.7.66:53756] AH01964: Connection to child 199 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.200257 2022] [ssl:info] [pid 11620:tid 140375417497344] [client ip.lan.12.42:65004] AH01964: Connection to child 192 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.308358 2022] [ssl:info] [pid 11620:tid 140374630725376] [client ip.lan.7.66:53757] AH01964: Connection to child 211 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.310417 2022] [ssl:info] [pid 11620:tid 140374597154560] [client ip.lan.12.42:65005] AH01964: Connection to child 215 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.359355 2022] [ssl:info] [pid 11396:tid 140373951244032] [client ip.lan.12.42:65006] AH01964: Connection to child 148 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.425288 2022] [ssl:info] [pid 11395:tid 140375383926528] [client ip.lan.7.66:53758] AH01964: Connection to child 68 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.471785 2022] [ssl:info] [pid 11395:tid 140375392319232] [client ip.lan.7.66:53759] AH01964: Connection to child 67 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:50.904093 2022] [ssl:info] [pid 11620:tid 140374949484288] [client ip.lan.1.23:58916] AH01964: Connection to child 206 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.091156 2022] [ssl:info] [pid 11620:tid 140374605547264] [client ip.lan.1.23:58918] AH01964: Connection to child 214 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.303362 2022] [ssl:info] [pid 11620:tid 140374924306176] [client ip.lan.1.23:58920] AH01964: Connection to child 209 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.361652 2022] [ssl:info] [pid 11620:tid 140375117240064] [client ip.lan.12.64:57158] AH01964: Connection to child 203 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.465884 2022] [ssl:info] [pid 11620:tid 140374966269696] [client ip.lan.1.23:58922] AH01964: Connection to child 204 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.473077 2022] [ssl:info] [pid 11620:tid 140375409104640] [client ip.lan.12.64:57159] AH01964: Connection to child 193 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.502463 2022] [ssl:info] [pid 11396:tid 140374488114944] [client ip.lan.1.29:59768] AH01964: Connection to child 141 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.570921 2022] [ssl:info] [pid 11396:tid 140374479722240] [client ip.lan.12.64:57160] AH01964: Connection to child 142 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.616658 2022] [ssl:info] [pid 11395:tid 140374320326400] [client ip.lan.12.64:57161] AH01964: Connection to child 88 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.680510 2022] [ssl:info] [pid 11394:tid 140374832051968] [client ip.lan.1.47:48536] AH01964: Connection to child 4 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.686009 2022] [ssl:info] [pid 11620:tid 140374613939968] [client ip.lan.1.29:59770] AH01964: Connection to child 213 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.790779 2022] [ssl:info] [pid 11620:tid 140374932698880] [client ip.lan.12.25:40653] AH01964: Connection to child 208 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.850588 2022] [ssl:info] [pid 11620:tid 140374915913472] [client ip.lan.1.47:48538] AH01964: Connection to child 210 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:51.921201 2022] [ssl:info] [pid 11620:tid 140375400711936] [client ip.lan.1.29:59772] AH01964: Connection to child 194 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.055722 2022] [ssl:info] [pid 11620:tid 140375417497344] [client ip.lan.1.47:48540] AH01964: Connection to child 192 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.106505 2022] [ssl:info] [pid 11620:tid 140374597154560] [client ip.lan.1.29:59774] AH01964: Connection to child 215 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.127947 2022] [ssl:info] [pid 11620:tid 140375392319232] [client ip.lan.1.112:42162] AH01964: Connection to child 195 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.230496 2022] [ssl:info] [pid 11620:tid 140375167596288] [client ip.lan.1.47:48542] AH01964: Connection to child 197 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.366956 2022] [ssl:info] [pid 11620:tid 140374605547264] [client ip.lan.1.112:42164] AH01964: Connection to child 214 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.548723 2022] [ssl:info] [pid 11620:tid 140375117240064] [client ip.lan.7.64:51037] AH01964: Connection to child 203 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.573199 2022] [ssl:info] [pid 11395:tid 140375409104640] [client ip.lan.3.12:1971] AH01964: Connection to child 65 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.612464 2022] [ssl:info] [pid 11620:tid 140374924306176] [client ip.lan.3.12:1972] AH01964: Connection to child 209 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.612819 2022] [ssl:info] [pid 11620:tid 140374588761856] [client ip.lan.1.10:50180] AH01964: Connection to child 216 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.660924 2022] [ssl:info] [pid 11620:tid 140375159203584] [client ip.lan.1.112:42166] AH01964: Connection to child 198 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.686165 2022] [ssl:info] [pid 11620:tid 140375142418176] [client ip.lan.7.64:51038] AH01964: Connection to child 200 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.692994 2022] [ssl:info] [pid 11620:tid 140374613939968] [client ip.lan.3.12:1973] AH01964: Connection to child 213 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.746637 2022] [ssl:info] [pid 11620:tid 140374932698880] [client ip.lan.3.12:1974] AH01964: Connection to child 208 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.807891 2022] [ssl:info] [pid 11620:tid 140375400711936] [client ip.lan.1.10:50182] AH01964: Connection to child 194 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.820621 2022] [ssl:info] [pid 11620:tid 140375134025472] [client ip.lan.6.202:59422] AH01964: Connection to child 201 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.821309 2022] [ssl:info] [pid 11620:tid 140375417497344] [client ip.lan.7.64:51039] AH01964: Connection to child 192 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.896778 2022] [ssl:info] [pid 11620:tid 140375392319232] [client ip.lan.7.64:51040] AH01964: Connection to child 195 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:52.939490 2022] [ssl:info] [pid 11620:tid 140375383926528] [client ip.lan.6.202:59423] AH01964: Connection to child 196 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.000952 2022] [ssl:info] [pid 11396:tid 140374999807744] [client ip.lan.7.64:51041] AH01964: Connection to child 137 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.022801 2022] [ssl:info] [pid 11620:tid 140375117240064] [client ip.lan.1.10:50184] AH01964: Connection to child 203 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.057884 2022] [ssl:info] [pid 11620:tid 140374605547264] [client ip.lan.7.64:51042] AH01964: Connection to child 214 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.059036 2022] [ssl:info] [pid 11620:tid 140375409104640] [client ip.lan.6.202:59424] AH01964: Connection to child 193 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.083903 2022] [ssl:info] [pid 11620:tid 140374924306176] [client ip.lan.1.112:42168] AH01964: Connection to child 209 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.123412 2022] [ssl:info] [pid 11396:tid 140375392319232] [client ip.lan.6.202:59425] AH01964: Connection to child 131 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.186530 2022] [ssl:info] [pid 11620:tid 140375142418176] [client ip.lan.1.10:50186] AH01964: Connection to child 200 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.336855 2022] [ssl:info] [pid 11620:tid 140374932698880] [client ip.lan.1.112:42170] AH01964: Connection to child 208 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.622871 2022] [ssl:info] [pid 11620:tid 140374941091584] [client ip.lan.1.112:42172] AH01964: Connection to child 207 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.855829 2022] [ssl:info] [pid 11620:tid 140374630725376] [client ip.lan.1.112:42174] AH01964: Connection to child 211 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:53.937131 2022] [ssl:info] [pid 11394:tid 140374823659264] [client ip.lan.1.64:36478] AH01964: Connection to child 5 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:54.109148 2022] [ssl:info] [pid 11620:tid 140374597154560] [client ip.lan.1.64:36480] AH01964: Connection to child 215 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:54.310399 2022] [ssl:info] [pid 11620:tid 140374949484288] [client ip.lan.1.64:36482] AH01964: Connection to child 206 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:54.481079 2022] [ssl:info] [pid 11620:tid 140375167596288] [client ip.lan.1.64:36484] AH01964: Connection to child 197 established (server ip.lan.4.1:443) [Fri Jan 14 08:46:55.657860 2022] [ssl:info] [pid 11394:tid 140373817026304] [client ip.lan.7.62:52485] AH01964: Connection to child 19 established (server ip.lan.4.1:443)
And I steel have a lot of waiting waiting process:
[root@fogus ~]# netstat -nat |grep 9000 |wc -l
850 -
@lebrun78 Also take a look at the Apache access.log. Check if you see clients constantly trying to download an update (
SmartInstaller.exe
). -
@sebastian-roth
Here is the access.log:ip.lan.1.41 - - [14/Jan/2022:14:40:09 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:04:B0:BB%7C14:B3:1F:04:B0:BB%7C14:B3:1F:04:B0:BB%7CFE:11:E3:62:00:00%7C02:42:6A:7F:73:A2&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.1.65 - - [14/Jan/2022:14:40:09 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.1.122 - - [14/Jan/2022:14:40:09 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.41 - - [14/Jan/2022:14:40:09 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.1.65 - - [14/Jan/2022:14:40:09 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.3.81 - - [14/Jan/2022:14:40:09 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 141 "-" "-" ip.lan.1.41 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.3.81 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:48:91%7C14:B3:1F:02:48:91%7C14:B3:1F:02:48:91%7C02:42:F2:07:9B:73&newService&json HTTP/1.1" 200 841 "-" "-" ip.lan.3.81 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.1.114 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 141 "-" "-" ip.lan.3.48 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 141 "-" "-" ip.lan.3.81 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.114 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=78:AC:C0:B1:AB:2E%7C76:7F:5C:90:EF:0A%7C02:42:F9:20:A5:67&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.3.48 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:8A:29%7C14:B3:1F:02:8A:29%7C14:B3:1F:02:8A:29%7C02:42:67:E3:1C:6C&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.3.48 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.1.114 - - [14/Jan/2022:14:40:10 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.48 - - [14/Jan/2022:14:40:11 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.114 - - [14/Jan/2022:14:40:11 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.7.11 - - [14/Jan/2022:14:40:11 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.7.11 - - [14/Jan/2022:14:40:11 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=74:E6:E2:DA:C1:2C&newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.7.61 - - [14/Jan/2022:14:40:11 +0100] "GET /fog/management/other/ssl/srvpublic.crt HTTP/1.1" 200 1740 "-" "-" ip.lan.3.103 - - [14/Jan/2022:14:40:11 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 141 "-" "-" ip.lan.7.61 - - [14/Jan/2022:14:40:11 +0100] "POST /fog/management/index.php?sub=requestClientInfo&authorize&newService HTTP/1.1" 200 15 "-" "-" ip.lan.3.103 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=B8:85:84:C5:BA:0E&newService&json HTTP/1.1" 200 809 "-" "-" ip.lan.7.61 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.103 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.7.61 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.3.103 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.67 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.1.67 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:4E:F6%7C14:B3:1F:02:4E:F6%7C14:B3:1F:02:4E:F6%7CFE:11:E7:61:00:34%7C02:42:30:08:E2:16&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.1.67 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.1.67 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.103 - - [14/Jan/2022:14:40:12 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.1.103 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=78:AC:C0:AF:BE:3F%7C42:14:34:62:41:59%7C02:42:07:3A:5E:1C&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.3.106 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 141 "-" "-" ip.lan.12.64 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.3.106 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=E4:B9:7A:F1:33:DC&newService&json HTTP/1.1" 200 809 "-" "-" ip.lan.1.103 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.106 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.106 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.12.64 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=0A:00:27:00:00:0C%7C14:B3:1F:19:82:1D%7C00:50:56:C0:00:01%7C00:50:56:C0:00:08&newService&json HTTP/1.1" 200 841 "-" "-" ip.lan.12.64 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.12.64 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.103 - - [14/Jan/2022:14:40:13 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.7 - - [14/Jan/2022:14:40:15 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.5.12 - - [14/Jan/2022:14:40:15 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.1.7 - - [14/Jan/2022:14:40:15 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:4F:E9%7C14:B3:1F:02:4F:E9%7C14:B3:1F:02:4F:E9%7CFE:11:E7:61:01:54%7C02:42:93:0B:C7:73&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.5.12 - - [14/Jan/2022:14:40:15 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=9C:8E:99:F5:68:BF%7C0E:EA:59:3A:BB:36%7C02:42:1F:87:5F:23&newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.7 - - [14/Jan/2022:14:40:15 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.5.12 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/management/other/ssl/srvpublic.crt HTTP/1.1" 200 1740 "-" "-" ip.lan.1.7 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.3.22 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.3.11 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 141 "-" "-" ip.lan.3.22 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=E4:B9:7A:F1:2A:DF&newService&json HTTP/1.1" 200 713 "-" "-" ip.lan.3.22 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.11 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=E4:B9:7A:F1:2B:23&newService&json HTTP/1.1" 200 713 "-" "-" ip.lan.3.22 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.5.12 - - [14/Jan/2022:14:40:16 +0100] "POST /fog/management/index.php?sub=requestClientInfo&authorize&newService HTTP/1.1" 200 326 "-" "-" ip.lan.3.11 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.11 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.5.12 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=9C:8E:99:F5:68:BF%7C0E:EA:59:3A:BB:36%7C02:42:1F:87:5F:23&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.5.12 - - [14/Jan/2022:14:40:16 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.5.12 - - [14/Jan/2022:14:40:17 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.6.72 - - [14/Jan/2022:14:40:17 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.6.72 - - [14/Jan/2022:14:40:18 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:75:0B%7C14:B3:1F:02:75:0B%7C14:B3:1F:02:75:0B%7C02:42:F2:20:16:E9&newService&json HTTP/1.1" 200 809 "-" "-" ip.lan.6.72 - - [14/Jan/2022:14:40:18 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.6.72 - - [14/Jan/2022:14:40:18 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.3.88 - - [14/Jan/2022:14:40:19 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.3.88 - - [14/Jan/2022:14:40:19 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:01:C2:54%7C14:B3:1F:01:C2:54%7C14:B3:1F:01:C2:54%7C02:42:42:16:8A:D1&newService&json HTTP/1.1" 200 841 "-" "-" ip.lan.3.88 - - [14/Jan/2022:14:40:19 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.88 - - [14/Jan/2022:14:40:19 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.33 - - [14/Jan/2022:14:40:19 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 141 "-" "-" ip.lan.6.70 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 141 "-" "-" ip.lan.1.33 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:48:A9%7C14:B3:1F:02:48:A9%7C14:B3:1F:02:48:A9%7CFE:11:E3:62:00:02%7C02:42:F5:A3:8B:48&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.3.37 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.6.70 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:01:0C:11%7C14:B3:1F:01:0C:11%7C14:B3:1F:01:0C:11%7C02:42:E5:32:78:6B&newService&json HTTP/1.1" 200 809 "-" "-" ip.lan.1.33 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.37 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:50:03%7C14:B3:1F:02:50:03%7C14:B3:1F:02:50:03%7C02:42:A0:AE:A0:52&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.6.70 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.1.33 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.3.37 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.6.70 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.3.87 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.3.37 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.3.32 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.3.87 - - [14/Jan/2022:14:40:20 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:48:D7%7C14:B3:1F:02:48:D7%7C14:B3:1F:02:48:D7%7C02:42:CE:55:48:81&newService&json HTTP/1.1" 200 841 "-" "-" ip.lan.3.32 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:50:7A%7C14:B3:1F:02:50:7A%7C14:B3:1F:02:50:7A%7C02:42:5E:15:79:9F&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.3.87 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.32 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.3.87 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.3.32 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.34 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 141 "-" "-" ip.lan.1.1 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/management/index.php?sub=requestClientInfo&configure&newService&json HTTP/1.1" 200 142 "-" "-" ip.lan.1.34 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:01:C1:34%7C14:B3:1F:01:C1:34%7C14:B3:1F:01:C1:34%7CFE:11:E7:61:00:41%7C02:42:A9:06:FE:3A&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.1.1 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/management/index.php?sub=requestClientInfo&mac=14:B3:1F:02:48:80%7C14:B3:1F:02:48:80%7C14:B3:1F:02:48:80%7CFE:11:E7:61:01:92&newService&json HTTP/1.1" 200 745 "-" "-" ip.lan.1.34 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.1.1 - - [14/Jan/2022:14:40:21 +0100] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 7 "-" "-" ip.lan.1.34 - - [14/Jan/2022:14:40:22 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-" ip.lan.1.1 - - [14/Jan/2022:14:40:22 +0100] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 5 "-" "-"
-
@lebrun78 said in FOG Multicast Not starting anymore:
Here is the access.log:
Ok, good it’s not an update loop. That’s important to know. So we are back to what George was onto with switching to InnoDB. In theory this should enable heaps more clients to query information from the database in parallel than before. But somehow looks like this was not actually the bottleneck on your system.
There are several topics here in the forums discussing performance and load:
https://forums.fogproject.org/topic/14254/high-cpu-usage
https://forums.fogproject.org/topic/12392/fog-server-high-cpu
https://forums.fogproject.org/topic/14689/high-cpu-usage-on-fog-server
…@george1421 Do you think tweaking PHP-FPM paramaters will help any further?
It’s a lot of work for devs to simulate high loads with several hundreds of fog-client machines. So this depends on people like you to be as keen to dig into it.
-
@sebastian-roth said in FOG Multicast Not starting anymore:
@george1421 Do you think tweaking PHP-FPM paramaters will help any further?
Right now I’m of the opinion the FOG server needs more ram since swap space is being used. But I don’t think I’ve been answered yet on how many computers with the FOG Client installed and if the php-fpm settings have been tweaked from the FOG defaults. If we are talking about over 2000 computers we need to do some testing.
lebrun78 was able to provide some pre-innodb database upgrade statistics where it looks like we can optimize the database searches with some smartly created indexes. I didn’t want to go into that until he system returned to a normal state.
-
@george1421 said:
But I don’t think I’ve been answered yet on how many computers with the FOG Client installed and if the php-fpm settings have been tweaked from the FOG defaults.
Within the SQL query log there is a list of hosts. That has about 500 entries.
Also I am quoting @lebrun78 post in this same forum topic some days ago:
I have more than 300 computer deployed with Fog so I collect info that you need.
-
Geoges, should I use this topic as a reference ?
https://forums.fogproject.org/topic/10717/can-php-fpm-make-fog-web-gui-fast/2 -
@lebrun78 I don’t have hard evidence but I have doubts about your server. You should not have full memory usage with 300 client computers (guess). So I’m wondering if you made changes to the php-fpm settings to solve another problem. (??).
Search the /etc directory for a file called www.conf. In there I want to know if you adjusted any of these values
pm.max_children = 50 pm.start_servers = 5 pm.min_spare_servers = 5 pm.max_spare_servers = 35
or this value
php_admin_value[memory_limit] = 256M
Can you tell us more about the hardware your FOG server is running on? (Manufacturer and model) What about your hard drives on your FOG server, is it a VM disk, raid disk array or a single sata disk? Are you running any other programs on the fog server other than FOG?
You can reduce some of the load on your fog server by changing the FOG Client check in time from 5 minutes (default) to 10 or 15 minutes. This will slow down the number of clients that check in per minute to give your FOG server a chance to keep up.
-
@george1421
changed to pm.max_spare_servers = 35
other paramaters had the correct valueThe hardware :
PowerEdge R440
Intel Xeon Silver 4208 CPU @ 2.10GHz
memory size: 16GiB
network BCM57412 NetXtreme-E 10Gb