MYSQL/HTTPD resource issues in 5020
-
@Adam-Taylor Do you have a lot of clients? Are you using FOG_HOST_LOOKUP? (On hosts do you see the green/red dots when on list/search?)
-
@Adam-Taylor Please install the packet
lsofon your FOG server and runlsof -i :80to see who keeps all those apache processes busy. -
Right now i have 33525 mysql connections when i run “show status like ‘Con%’;” in mysql cli. And no…that was not a mistype
The connections are just not going away.Total registered hosts are 1892 of which about 1800 have the fog client.
Running lsof, I am a around 20-30 established connections at any one given time with about the same listening. All while the CPU is around 100%. Every once and a while it might spike to around 180 or so but then drop back down within 30 seconds or so.
FOG_HOST_LOOKUP was checked and I just unchecked it to see if that helps. So far I am not seeing a difference after a total system reboot.
I had to restart the mysqld process so that I could get to that setting as the web gui would not do anything. In that 2 minutes from a reboot, I am back upto 297 mysql connections.
BTW, I am running this on RedHat Enterprise 6.
Thanks!
Adam
-
What does
lsof -i :3306show? Is it all localhost connections?And what about
lsof -i :80? All connections from different clients?? -
Hmm, i’m getting this now too! FOG server is being crippled even after a reboot and wiping the hosts/reimporting (and Host lookup isn’t checked)
I am assuming that its because we have lots of PCs checking in but I didn’t think that so many processes would get generated!
Also lsof for port 80:
root@fog:/var/log/mysql# lsof -i :80 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME apache2 2084 root 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 2636 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 2636 www-data 16u IPv6 205385 0t0 TCP fog.myserver.co.uk:http->10.1.24.175:49760 (ESTABLISHED) apache2 2768 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 2768 www-data 16u IPv6 211218 0t0 TCP fog.myserver.co.uk:http->10.28.10.238:49570 (ESTABLISHED) apache2 2807 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 2807 www-data 16u IPv6 206891 0t0 TCP fog.myserver.co.uk:http->10.9.110.159:57368 (ESTABLISHED) apache2 2818 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 2818 www-data 16u IPv6 208587 0t0 TCP fog.myserver.co.uk:http->10.1.26.98:49835 (ESTABLISHED) apache2 2863 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 2863 www-data 16u IPv6 210004 0t0 TCP fog.myserver.co.uk:http->10.28.10.167:53887 (ESTABLISHED) apache2 3093 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 3093 www-data 16u IPv6 211334 0t0 TCP fog.myserver.co.uk:http->10.1.24.122:65430 (ESTABLISHED) apache2 3097 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 3097 www-data 16u IPv6 211239 0t0 TCP fog.myserver.co.uk:http->10.1.21.170:29595 (ESTABLISHED) apache2 3579 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 3579 www-data 16u IPv6 208959 0t0 TCP fog.myserver.co.uk:http->10.9.110.108:56545 (ESTABLISHED) apache2 3617 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 3617 www-data 16u IPv6 208904 0t0 TCP fog.myserver.co.uk:http->10.9.112.158:64571 (ESTABLISHED) apache2 3657 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 3657 www-data 16u IPv6 207246 0t0 TCP fog.myserver.co.uk:http->10.9.110.166:58779 (ESTABLISHED) apache2 4124 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4124 www-data 16u IPv6 205230 0t0 TCP fog.myserver.co.uk:http->10.28.10.193:65050 (ESTABLISHED) apache2 4134 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4134 www-data 16u IPv6 207383 0t0 TCP fog.myserver.co.uk:http->10.240.11.209:60319 (ESTABLISHED) apache2 4160 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4160 www-data 16u IPv6 207332 0t0 TCP fog.myserver.co.uk:http->10.9.112.200:53346 (ESTABLISHED) apache2 4176 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4176 www-data 16u IPv6 204956 0t0 TCP fog.myserver.co.uk:http->10.28.10.21:49847 (ESTABLISHED) apache2 4190 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4190 www-data 16u IPv6 206864 0t0 TCP fog.myserver.co.uk:http->10.1.25.145:54472 (ESTABLISHED) apache2 4191 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4191 www-data 16u IPv6 208675 0t0 TCP fog.myserver.co.uk:http->10.1.26.148:60850 (ESTABLISHED) apache2 4205 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4205 www-data 16u IPv6 207418 0t0 TCP fog.myserver.co.uk:http->10.1.25.146:56839 (ESTABLISHED) apache2 4220 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4220 www-data 16u IPv6 208115 0t0 TCP fog.myserver.co.uk:http->pc24-6.myserver.co.uk:57599 (ESTABLISHED) apache2 4225 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4225 www-data 16u IPv6 209198 0t0 TCP fog.myserver.co.uk:http->10.9.106.35:49624 (ESTABLISHED) apache2 4669 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4669 www-data 16u IPv6 209862 0t0 TCP fog.myserver.co.uk:http->10.28.10.68:49459 (ESTABLISHED) apache2 4672 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4672 www-data 16u IPv6 209534 0t0 TCP fog.myserver.co.uk:http->10.240.11.198:49391 (ESTABLISHED) apache2 4675 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4675 www-data 16u IPv6 207291 0t0 TCP fog.myserver.co.uk:http->10.28.10.166:54133 (ESTABLISHED) apache2 4691 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4691 www-data 16u IPv6 207738 0t0 TCP fog.myserver.co.uk:http->10.28.10.21:49848 (ESTABLISHED) apache2 4694 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4694 www-data 16u IPv6 211263 0t0 TCP fog.myserver.co.uk:http->10.240.11.182:55943 (ESTABLISHED) apache2 4696 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4696 www-data 16u IPv6 209160 0t0 TCP fog.myserver.co.uk:http->10.1.23.237:56832 (ESTABLISHED) apache2 4735 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4735 www-data 16u IPv6 209401 0t0 TCP fog.myserver.co.uk:http->10.28.10.217:49597 (ESTABLISHED) apache2 4756 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4756 www-data 16u IPv6 209628 0t0 TCP fog.myserver.co.uk:http->10.240.11.162:64878 (ESTABLISHED) apache2 4768 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4768 www-data 16u IPv6 206502 0t0 TCP fog.myserver.co.uk:http->10.9.106.41:49645 (FIN_WAIT2) apache2 4777 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 4777 www-data 16u IPv6 209994 0t0 TCP fog.myserver.co.uk:http->10.9.106.26:50253 (ESTABLISHED) apache2 5100 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5100 www-data 16u IPv6 209727 0t0 TCP fog.myserver.co.uk:http->10.1.25.173:56715 (ESTABLISHED) apache2 5128 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5128 www-data 16u IPv6 207472 0t0 TCP fog.myserver.co.uk:http->10.28.10.213:53558 (ESTABLISHED) apache2 5520 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5520 www-data 16u IPv6 209087 0t0 TCP fog.myserver.co.uk:http->10.1.23.231:51926 (ESTABLISHED) apache2 5530 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5530 www-data 16u IPv6 211051 0t0 TCP fog.myserver.co.uk:http->10.240.11.162:64879 (ESTABLISHED) apache2 5533 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5533 www-data 16u IPv6 205433 0t0 TCP fog.myserver.co.uk:http->10.1.37.27:27514 (ESTABLISHED) apache2 5544 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5544 www-data 16u IPv6 205271 0t0 TCP fog.myserver.co.uk:http->10.1.23.148:49740 (ESTABLISHED) apache2 5551 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5551 www-data 16u IPv6 208381 0t0 TCP fog.myserver.co.uk:http->10.9.112.189:49842 (ESTABLISHED) apache2 5570 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5570 www-data 16u IPv6 211113 0t0 TCP fog.myserver.co.uk:http->10.9.106.40:49619 (ESTABLISHED) apache2 5572 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5572 www-data 16u IPv6 208036 0t0 TCP fog.myserver.co.uk:http->10.9.110.128:65140 (ESTABLISHED) apache2 5580 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5580 www-data 16u IPv6 206913 0t0 TCP fog.myserver.co.uk:http->10.1.23.230:49844 (ESTABLISHED) apache2 5602 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5602 www-data 16u IPv6 208805 0t0 TCP fog.myserver.co.uk:http->10.1.24.126:55469 (ESTABLISHED) apache2 5618 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5618 www-data 16u IPv6 206700 0t0 TCP fog.myserver.co.uk:http->10.9.106.29:49608 (ESTABLISHED) apache2 5619 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 5619 www-data 16u IPv6 208974 0t0 TCP fog.myserver.co.uk:http->10.1.55.236:49938 (ESTABLISHED) apache2 6006 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6006 www-data 16u IPv6 209326 0t0 TCP fog.myserver.co.uk:http->10.1.25.118:55158 (ESTABLISHED) apache2 6268 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6268 www-data 16u IPv6 209858 0t0 TCP fog.myserver.co.uk:http->10.1.26.123:53330 (ESTABLISHED) apache2 6269 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6269 www-data 16u IPv6 211026 0t0 TCP fog.myserver.co.uk:http->10.28.12.239:52159 (ESTABLISHED) apache2 6270 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6270 www-data 16u IPv6 211175 0t0 TCP fog.myserver.co.uk:http->10.28.10.93:64841 (ESTABLISHED) apache2 6273 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6273 www-data 16u IPv6 209592 0t0 TCP fog.myserver.co.uk:http->10.9.110.127:58944 (ESTABLISHED) apache2 6292 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6292 www-data 16u IPv6 207455 0t0 TCP fog.myserver.co.uk:http->10.1.21.151:51066 (ESTABLISHED) apache2 6296 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6296 www-data 16u IPv6 209338 0t0 TCP fog.myserver.co.uk:http->10.1.24.160:52020 (ESTABLISHED) apache2 6304 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6304 www-data 16u IPv6 211197 0t0 TCP fog.myserver.co.uk:http->10.9.112.168:54211 (ESTABLISHED) apache2 6305 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6305 www-data 16u IPv6 211145 0t0 TCP fog.myserver.co.uk:http->10.28.11.210:49781 (ESTABLISHED) apache2 6787 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6787 www-data 16u IPv6 209787 0t0 TCP fog.myserver.co.uk:http->pc26-36.myserver.co.uk:49781 (ESTABLISHED) apache2 6794 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6794 www-data 16u IPv6 208319 0t0 TCP fog.myserver.co.uk:http->10.1.23.233:49922 (ESTABLISHED) apache2 6796 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 6796 www-data 16u IPv6 199442 0t0 TCP fog.myserver.co.uk:http->10.240.11.176:50216 (ESTABLISHED) apache2 14670 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14670 www-data 16u IPv6 208028 0t0 TCP fog.myserver.co.uk:http->10.1.25.150:58961 (ESTABLISHED) apache2 14671 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14671 www-data 16u IPv6 209766 0t0 TCP fog.myserver.co.uk:http->10.28.10.168:60393 (ESTABLISHED) apache2 14672 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14672 www-data 16u IPv6 205388 0t0 TCP fog.myserver.co.uk:http->10.1.23.211:49845 (ESTABLISHED) apache2 14673 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14673 www-data 16u IPv6 211298 0t0 TCP fog.myserver.co.uk:http->10.240.11.236:55911 (ESTABLISHED) apache2 14681 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14681 www-data 16u IPv6 209099 0t0 TCP fog.myserver.co.uk:http->10.1.26.103:51299 (ESTABLISHED) apache2 14682 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14682 www-data 16u IPv6 205304 0t0 TCP fog.myserver.co.uk:http->10.9.112.196:49838 (ESTABLISHED) apache2 14684 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14684 www-data 16u IPv6 211173 0t0 TCP fog.myserver.co.uk:http->10.1.23.209:50288 (ESTABLISHED) apache2 14685 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14685 www-data 16u IPv6 209595 0t0 TCP fog.myserver.co.uk:http->10.1.24.103:49763 (ESTABLISHED) apache2 14686 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14686 www-data 16u IPv6 208799 0t0 TCP fog.myserver.co.uk:http->10.28.10.222:51282 (ESTABLISHED) apache2 14687 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14687 www-data 16u IPv6 208014 0t0 TCP fog.myserver.co.uk:http->10.1.26.194:49692 (ESTABLISHED) apache2 14694 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14694 www-data 16u IPv6 206769 0t0 TCP fog.myserver.co.uk:http->10.1.23.205:51748 (ESTABLISHED) apache2 14696 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14696 www-data 16u IPv6 209771 0t0 TCP fog.myserver.co.uk:http->10.1.26.163:64329 (ESTABLISHED) apache2 14697 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14697 www-data 16u IPv6 207285 0t0 TCP fog.myserver.co.uk:http->10.1.25.174:56202 (ESTABLISHED) apache2 14701 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14701 www-data 16u IPv6 207548 0t0 TCP fog.myserver.co.uk:http->10.28.7.15:63067 (ESTABLISHED) apache2 14710 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14710 www-data 16u IPv6 209927 0t0 TCP fog.myserver.co.uk:http->10.28.10.178:59324 (ESTABLISHED) apache2 14713 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14713 www-data 16u IPv6 209794 0t0 TCP fog.myserver.co.uk:http->10.28.8.20:56839 (ESTABLISHED) apache2 14714 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14714 www-data 16u IPv6 211325 0t0 TCP fog.myserver.co.uk:http->10.28.10.180:49471 (ESTABLISHED) apache2 14716 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14716 www-data 16u IPv6 207539 0t0 TCP fog.myserver.co.uk:http->10.9.106.221:49627 (ESTABLISHED) apache2 14725 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14725 www-data 16u IPv6 209585 0t0 TCP fog.myserver.co.uk:http->pc24-17.myserver.co.uk:65308 (ESTABLISHED) apache2 14726 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14726 www-data 16u IPv6 199604 0t0 TCP fog.myserver.co.uk:http->10.1.25.139:65317 (ESTABLISHED) apache2 14727 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14727 www-data 16u IPv6 215258 0t0 TCP fog.myserver.co.uk:http->10.1.21.170:29597 (ESTABLISHED) apache2 14728 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14728 www-data 16u IPv6 209451 0t0 TCP fog.myserver.co.uk:http->10.1.23.219:51341 (ESTABLISHED) apache2 14731 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14731 www-data 16u IPv6 209752 0t0 TCP fog.myserver.co.uk:http->10.9.106.32:49646 (ESTABLISHED) apache2 14734 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14734 www-data 16u IPv6 206560 0t0 TCP fog.myserver.co.uk:http->10.1.25.143:50171 (ESTABLISHED) apache2 14735 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14735 www-data 16u IPv6 208701 0t0 TCP fog.myserver.co.uk:http->10.9.106.26:50250 (ESTABLISHED) apache2 14748 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14748 www-data 16u IPv6 206878 0t0 TCP fog.myserver.co.uk:http->10.1.25.135:53350 (ESTABLISHED) apache2 14756 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14756 www-data 16u IPv6 205462 0t0 TCP fog.myserver.co.uk:http->10.240.11.232:49580 (ESTABLISHED) apache2 14757 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 14757 www-data 16u IPv6 209305 0t0 TCP fog.myserver.co.uk:http->10.240.11.36:49789 (ESTABLISHED) apache2 15211 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15211 www-data 16u IPv6 205319 0t0 TCP fog.myserver.co.uk:http->10.28.10.194:62332 (ESTABLISHED) apache2 15234 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15234 www-data 16u IPv6 199457 0t0 TCP fog.myserver.co.uk:http->10.9.106.45:50690 (ESTABLISHED) apache2 15235 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15235 www-data 16u IPv6 208384 0t0 TCP fog.myserver.co.uk:http->10.1.26.171:50388 (ESTABLISHED) apache2 15282 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15282 www-data 16u IPv6 205353 0t0 TCP fog.myserver.co.uk:http->10.9.106.31:49660 (ESTABLISHED) apache2 15307 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15307 www-data 16u IPv6 199602 0t0 TCP fog.myserver.co.uk:http->10.28.11.207:52033 (ESTABLISHED) apache2 15308 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15308 www-data 16u IPv6 204923 0t0 TCP fog.myserver.co.uk:http->10.9.112.166:56260 (ESTABLISHED) apache2 15352 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15352 www-data 16u IPv6 211225 0t0 TCP fog.myserver.co.uk:http->10.240.11.199:49886 (ESTABLISHED) apache2 15367 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15367 www-data 16u IPv6 209694 0t0 TCP fog.myserver.co.uk:http->pc26-25.myserver.co.uk:52138 (ESTABLISHED) apache2 15370 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15370 www-data 16u IPv6 209280 0t0 TCP fog.myserver.co.uk:http->10.1.25.156:53894 (ESTABLISHED) apache2 15371 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15371 www-data 16u IPv6 207943 0t0 TCP fog.myserver.co.uk:http->10.240.11.232:49564 (ESTABLISHED) apache2 15385 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15385 www-data 16u IPv6 206583 0t0 TCP fog.myserver.co.uk:http->10.9.106.36:49647 (ESTABLISHED) apache2 15389 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15389 www-data 16u IPv6 211236 0t0 TCP fog.myserver.co.uk:http->10.1.24.160:52021 (ESTABLISHED) apache2 15390 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15390 www-data 16u IPv6 209116 0t0 TCP fog.myserver.co.uk:http->10.1.23.197:50257 (ESTABLISHED) apache2 15393 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15393 www-data 16u IPv6 209989 0t0 TCP fog.myserver.co.uk:http->10.1.26.224:56221 (ESTABLISHED) apache2 15396 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15396 www-data 16u IPv6 209685 0t0 TCP fog.myserver.co.uk:http->10.28.10.24:55542 (ESTABLISHED) apache2 15399 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15399 www-data 16u IPv6 211314 0t0 TCP fog.myserver.co.uk:http->10.1.23.237:56843 (ESTABLISHED) apache2 15789 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15789 www-data 16u IPv6 208833 0t0 TCP fog.myserver.co.uk:http->10.28.7.35:49378 (ESTABLISHED) apache2 15790 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15790 www-data 16u IPv6 207953 0t0 TCP fog.myserver.co.uk:http->10.28.10.95:58542 (ESTABLISHED) apache2 15793 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15793 www-data 16u IPv6 206758 0t0 TCP fog.myserver.co.uk:http->10.1.26.105:50066 (ESTABLISHED) apache2 15794 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15794 www-data 16u IPv6 205274 0t0 TCP fog.myserver.co.uk:http->10.1.23.216:56813 (ESTABLISHED) apache2 15795 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15795 www-data 16u IPv6 206577 0t0 TCP fog.myserver.co.uk:http->10.1.26.94:62086 (ESTABLISHED) apache2 15803 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15803 www-data 16u IPv6 207925 0t0 TCP fog.myserver.co.uk:http->10.28.10.219:64208 (ESTABLISHED) apache2 15804 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15804 www-data 16u IPv6 209654 0t0 TCP fog.myserver.co.uk:http->10.240.11.36:49790 (ESTABLISHED) apache2 15806 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15806 www-data 16u IPv6 205341 0t0 TCP fog.myserver.co.uk:http->10.1.55.226:53041 (ESTABLISHED) apache2 15807 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15807 www-data 16u IPv6 208059 0t0 TCP fog.myserver.co.uk:http->10.9.106.34:49648 (ESTABLISHED) apache2 15808 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15808 www-data 16u IPv6 206831 0t0 TCP fog.myserver.co.uk:http->10.1.26.93:63211 (ESTABLISHED) apache2 15809 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15809 www-data 16u IPv6 209494 0t0 TCP fog.myserver.co.uk:http->10.28.10.203:55371 (FIN_WAIT2) apache2 15813 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15813 www-data 16u IPv6 207512 0t0 TCP fog.myserver.co.uk:http->10.28.10.93:49609 (ESTABLISHED) apache2 15814 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15814 www-data 16u IPv6 209625 0t0 TCP fog.myserver.co.uk:http->10.240.11.226:49533 (ESTABLISHED) apache2 15815 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15815 www-data 16u IPv6 199544 0t0 TCP fog.myserver.co.uk:http->10.1.26.88:49757 (ESTABLISHED) apache2 15816 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15816 www-data 16u IPv6 209879 0t0 TCP fog.myserver.co.uk:http->10.1.55.228:49738 (ESTABLISHED) apache2 15817 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15817 www-data 16u IPv6 208973 0t0 TCP fog.myserver.co.uk:http->10.28.10.168:60392 (ESTABLISHED) apache2 15818 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15818 www-data 16u IPv6 206574 0t0 TCP fog.myserver.co.uk:http->10.28.10.203:55369 (ESTABLISHED) apache2 15819 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15819 www-data 16u IPv6 210971 0t0 TCP fog.myserver.co.uk:http->10.28.12.225:50235 (ESTABLISHED) apache2 15820 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15820 www-data 16u IPv6 208821 0t0 TCP fog.myserver.co.uk:http->10.1.28.51:50042 (ESTABLISHED) apache2 15833 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15833 www-data 16u IPv6 208149 0t0 TCP fog.myserver.co.uk:http->10.28.10.11:54096 (ESTABLISHED) apache2 15837 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15837 www-data 16u IPv6 207433 0t0 TCP fog.myserver.co.uk:http->pc24-23.myserver.co.uk:50093 (ESTABLISHED) apache2 15927 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15927 www-data 16u IPv6 206566 0t0 TCP fog.myserver.co.uk:http->10.1.26.60:50475 (ESTABLISHED) apache2 15928 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15928 www-data 16u IPv6 205446 0t0 TCP fog.myserver.co.uk:http->10.28.10.157:63434 (ESTABLISHED) apache2 15929 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 15929 www-data 16u IPv6 206629 0t0 TCP fog.myserver.co.uk:http->10.28.10.199:54524 (ESTABLISHED) apache2 16107 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 16107 www-data 16u IPv6 208099 0t0 TCP fog.myserver.co.uk:http->10.28.10.14:50199 (ESTABLISHED) apache2 17032 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17032 www-data 16u IPv6 207327 0t0 TCP fog.myserver.co.uk:http->10.28.10.192:51399 (ESTABLISHED) apache2 17311 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17311 www-data 16u IPv6 207968 0t0 TCP fog.myserver.co.uk:http->10.240.11.153:49295 (ESTABLISHED) apache2 17315 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17315 www-data 16u IPv6 208474 0t0 TCP fog.myserver.co.uk:http->10.9.106.27:49633 (ESTABLISHED) apache2 17316 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17316 www-data 16u IPv6 206688 0t0 TCP fog.myserver.co.uk:http->10.9.110.165:56298 (ESTABLISHED) apache2 17320 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17320 www-data 16u IPv6 209074 0t0 TCP fog.myserver.co.uk:http->10.1.24.172:51523 (ESTABLISHED) apache2 17321 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17321 www-data 16u IPv6 208433 0t0 TCP fog.myserver.co.uk:http->10.1.25.151:50002 (ESTABLISHED) apache2 17322 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17322 www-data 16u IPv6 206784 0t0 TCP fog.myserver.co.uk:http->10.28.12.225:50233 (ESTABLISHED) apache2 17323 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17323 www-data 16u IPv6 207321 0t0 TCP fog.myserver.co.uk:http->10.1.25.178:58762 (ESTABLISHED) apache2 17328 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17328 www-data 16u IPv6 206844 0t0 TCP fog.myserver.co.uk:http->10.1.23.239:50334 (ESTABLISHED) apache2 17329 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17329 www-data 16u IPv6 199659 0t0 TCP fog.myserver.co.uk:http->10.240.11.235:49983 (ESTABLISHED) apache2 17330 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17330 www-data 16u IPv6 205477 0t0 TCP fog.myserver.co.uk:http->10.28.10.177:59575 (ESTABLISHED) apache2 17335 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17335 www-data 16u IPv6 208329 0t0 TCP fog.myserver.co.uk:http->10.1.25.129:52700 (ESTABLISHED) apache2 17336 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17336 www-data 16u IPv6 209230 0t0 TCP fog.myserver.co.uk:http->10.1.37.27:27513 (ESTABLISHED) apache2 17337 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 17337 www-data 16u IPv6 199409 0t0 TCP fog.myserver.co.uk:http->10.240.11.182:55925 (ESTABLISHED) apache2 18513 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18513 www-data 16u IPv6 208128 0t0 TCP fog.myserver.co.uk:http->10.1.26.166:63340 (ESTABLISHED) apache2 18519 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18519 www-data 16u IPv6 211274 0t0 TCP fog.myserver.co.uk:http->10.1.25.119:50908 (ESTABLISHED) apache2 18525 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18525 www-data 16u IPv6 207974 0t0 TCP fog.myserver.co.uk:http->10.240.11.199:49885 (ESTABLISHED) apache2 18526 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18526 www-data 16u IPv6 208623 0t0 TCP fog.myserver.co.uk:http->pc24-31.myserver.co.uk:49703 (ESTABLISHED) apache2 18530 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18530 www-data 16u IPv6 209554 0t0 TCP fog.myserver.co.uk:http->10.28.10.155:56252 (ESTABLISHED) apache2 18534 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18534 www-data 16u IPv6 205313 0t0 TCP fog.myserver.co.uk:http->10.28.10.60:52758 (ESTABLISHED) apache2 18535 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18535 www-data 16u IPv6 207876 0t0 TCP fog.myserver.co.uk:http->10.9.106.43:49624 (ESTABLISHED) apache2 18542 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18542 www-data 16u IPv6 211385 0t0 TCP fog.myserver.co.uk:http->10.28.11.223:50357 (ESTABLISHED) apache2 18555 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18555 www-data 16u IPv6 206477 0t0 TCP fog.myserver.co.uk:http->10.1.55.107:51872 (FIN_WAIT2) apache2 18560 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18560 www-data 16u IPv6 206707 0t0 TCP fog.myserver.co.uk:http->10.28.12.239:52158 (ESTABLISHED) apache2 18561 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18561 www-data 16u IPv6 209474 0t0 TCP fog.myserver.co.uk:http->10.1.26.166:63341 (FIN_WAIT2) apache2 18564 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18564 www-data 16u IPv6 206586 0t0 TCP fog.myserver.co.uk:http->10.28.7.60:49773 (ESTABLISHED) apache2 18565 www-data 4u IPv6 12460 0t0 TCP *:http (LISTEN) apache2 18565 www-data 16u IPv6 208156 0t0 TCP fog.myserver.co.uk:http->10.28.10.92:58723 (ESTABLISHED) root@fog:/var/log/mysql#Will keep an eye on what people do
-
I added the following to the MySQL.class.php file to see how often it was creating a SQL connection:
public function connect()
try
if (!$this->link)
file_put_contents(‘php://stderr’, print_r(DATABASE_HOST, TRUE))It seems to be getting called 7-10 times per second currently.
Every single connection is p:127.0.0.1 so every connection is persistant…and they are not getting closed would be my guess.
Thanks!
Adam
-
3306 is all localhost connections.
80 is all different clients from around the campus.
-
I changed the DB connect from persistant to non persistant to test and see if the connections would then go away.
The connections go up much faster then before (what i expected since no reuse) but once again…they never go down.
That’s all i know to look at/try at the moment. Do appreciate your help in this!
Thanks!
Adam
-
@Adam-Taylor I know I keep asking the same things, but please reupdate and try again?
The only other thing that I think could be causing the issue is in line and hopefully this is corrected for. I have no real clue whether or not it is the case though.
-
The system is still getting hammered and all four cores are at 100% after a complete reboot.
Adam
-
Also, that 100% is always ~30% for mysql and the rest is chewed up with httpd processes.
Adam
-
@Adam-Taylor Tom has made some changes, update again.
-
Changes don’t seem to have fixed anything for me. Still see 100% usage after restart and update to 5078- basically the same as my screenshot earlier.
I do see this in the apache error log though, for all the clients that are trying to connect. Not sure itd make any difference though.
:[Mon Oct 26 14:55:09.526015 2015] [:error] [pid 3099] [client 10.240.5.151:57133] PHP Warning: mysqli::mysqli(): (HY000/2002): No such file or directory in /var/www/html/fog/lib/db/MySQL.class.php on line 18 -
@Trevelyan this makes absolutely no sense. It sounds like, from the error you are reporting, your server doesn’t have the mysqlnd driver stack for php as it can’t even find the mysqli object.
-
I am still seeing 100% usage for me to after the last update with mysql using~30% and httpd using the rest with LOTS of processes.
Adam
-
@Adam-Taylor WHen I get back to a normal schedule at work, I’d love to link up and run a Teamviewer connection with you.
It seems, to me, that this isn’t something I’ve been causing, directly. For some time I had some issues, but I have a person who has around 6k hosts and was seeing the resource problems as described. I have confirmed with him that this is now fixed. Now that doesn’t mean an issue still doesn’t persist, but I’m pretty sure this isn’t something FOG is doing any more.
-
Can you ask him what OS, apache version, mysql version he is running for a can compare?
Thanks,
Adam
-
Question.
Would the old fog client be causing this? We noticed the new MSI fog client but we are still running the old one.
Adam
-
I’ve noticed that first thing in the morning, our server is fine. Its only when PCs start powering up that we see issues - but it isn’t even a hundred that cause it. Just loads and loads of high-CPU hogging apache2 instances that never seem to end.
For the record - all of these are new clients (we have no PCs running the old client anymore)
-
Are you both running the latest SVN? Would be very interesting to see what is going on within those lots of processes! Maybe you could try using
strace(option -p to attach to a running process, -o to write to a file) to see what is going on. I am aware of this being an advanced debugging issue and I am not sure if you are keen enough to get into this. Maybe give it a try and well see what we get. I am more than happy to take a look if you upload a strace log file.
