MYSQL/HTTPD resource issues in 5020



  • I added the following to the MySQL.class.php file to see how often it was creating a SQL connection:

    public function connect()
    try
    if (!$this->link)
    file_put_contents(‘php://stderr’, print_r(DATABASE_HOST, TRUE))

    It seems to be getting called 7-10 times per second currently.

    Every single connection is p:127.0.0.1 so every connection is persistant…and they are not getting closed would be my guess.

    Thanks!

    Adam



  • Hmm, i’m getting this now too! FOG server is being crippled even after a reboot and wiping the hosts/reimporting (and Host lookup isn’t checked)

    I am assuming that its because we have lots of PCs checking in but I didn’t think that so many processes would get generated!

    lolled.png

    Also lsof for port 80:

    root@fog:/var/log/mysql# lsof -i :80
    COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
    apache2  2084     root    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  2636 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  2636 www-data   16u  IPv6 205385      0t0  TCP fog.myserver.co.uk:http->10.1.24.175:49760 (ESTABLISHED)
    apache2  2768 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  2768 www-data   16u  IPv6 211218      0t0  TCP fog.myserver.co.uk:http->10.28.10.238:49570 (ESTABLISHED)
    apache2  2807 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  2807 www-data   16u  IPv6 206891      0t0  TCP fog.myserver.co.uk:http->10.9.110.159:57368 (ESTABLISHED)
    apache2  2818 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  2818 www-data   16u  IPv6 208587      0t0  TCP fog.myserver.co.uk:http->10.1.26.98:49835 (ESTABLISHED)
    apache2  2863 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  2863 www-data   16u  IPv6 210004      0t0  TCP fog.myserver.co.uk:http->10.28.10.167:53887 (ESTABLISHED)
    apache2  3093 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  3093 www-data   16u  IPv6 211334      0t0  TCP fog.myserver.co.uk:http->10.1.24.122:65430 (ESTABLISHED)
    apache2  3097 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  3097 www-data   16u  IPv6 211239      0t0  TCP fog.myserver.co.uk:http->10.1.21.170:29595 (ESTABLISHED)
    apache2  3579 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  3579 www-data   16u  IPv6 208959      0t0  TCP fog.myserver.co.uk:http->10.9.110.108:56545 (ESTABLISHED)
    apache2  3617 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  3617 www-data   16u  IPv6 208904      0t0  TCP fog.myserver.co.uk:http->10.9.112.158:64571 (ESTABLISHED)
    apache2  3657 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  3657 www-data   16u  IPv6 207246      0t0  TCP fog.myserver.co.uk:http->10.9.110.166:58779 (ESTABLISHED)
    apache2  4124 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4124 www-data   16u  IPv6 205230      0t0  TCP fog.myserver.co.uk:http->10.28.10.193:65050 (ESTABLISHED)
    apache2  4134 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4134 www-data   16u  IPv6 207383      0t0  TCP fog.myserver.co.uk:http->10.240.11.209:60319 (ESTABLISHED)
    apache2  4160 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4160 www-data   16u  IPv6 207332      0t0  TCP fog.myserver.co.uk:http->10.9.112.200:53346 (ESTABLISHED)
    apache2  4176 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4176 www-data   16u  IPv6 204956      0t0  TCP fog.myserver.co.uk:http->10.28.10.21:49847 (ESTABLISHED)
    apache2  4190 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4190 www-data   16u  IPv6 206864      0t0  TCP fog.myserver.co.uk:http->10.1.25.145:54472 (ESTABLISHED)
    apache2  4191 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4191 www-data   16u  IPv6 208675      0t0  TCP fog.myserver.co.uk:http->10.1.26.148:60850 (ESTABLISHED)
    apache2  4205 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4205 www-data   16u  IPv6 207418      0t0  TCP fog.myserver.co.uk:http->10.1.25.146:56839 (ESTABLISHED)
    apache2  4220 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4220 www-data   16u  IPv6 208115      0t0  TCP fog.myserver.co.uk:http->pc24-6.myserver.co.uk:57599 (ESTABLISHED)
    apache2  4225 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4225 www-data   16u  IPv6 209198      0t0  TCP fog.myserver.co.uk:http->10.9.106.35:49624 (ESTABLISHED)
    apache2  4669 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4669 www-data   16u  IPv6 209862      0t0  TCP fog.myserver.co.uk:http->10.28.10.68:49459 (ESTABLISHED)
    apache2  4672 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4672 www-data   16u  IPv6 209534      0t0  TCP fog.myserver.co.uk:http->10.240.11.198:49391 (ESTABLISHED)
    apache2  4675 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4675 www-data   16u  IPv6 207291      0t0  TCP fog.myserver.co.uk:http->10.28.10.166:54133 (ESTABLISHED)
    apache2  4691 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4691 www-data   16u  IPv6 207738      0t0  TCP fog.myserver.co.uk:http->10.28.10.21:49848 (ESTABLISHED)
    apache2  4694 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4694 www-data   16u  IPv6 211263      0t0  TCP fog.myserver.co.uk:http->10.240.11.182:55943 (ESTABLISHED)
    apache2  4696 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4696 www-data   16u  IPv6 209160      0t0  TCP fog.myserver.co.uk:http->10.1.23.237:56832 (ESTABLISHED)
    apache2  4735 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4735 www-data   16u  IPv6 209401      0t0  TCP fog.myserver.co.uk:http->10.28.10.217:49597 (ESTABLISHED)
    apache2  4756 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4756 www-data   16u  IPv6 209628      0t0  TCP fog.myserver.co.uk:http->10.240.11.162:64878 (ESTABLISHED)
    apache2  4768 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4768 www-data   16u  IPv6 206502      0t0  TCP fog.myserver.co.uk:http->10.9.106.41:49645 (FIN_WAIT2)
    apache2  4777 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  4777 www-data   16u  IPv6 209994      0t0  TCP fog.myserver.co.uk:http->10.9.106.26:50253 (ESTABLISHED)
    apache2  5100 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5100 www-data   16u  IPv6 209727      0t0  TCP fog.myserver.co.uk:http->10.1.25.173:56715 (ESTABLISHED)
    apache2  5128 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5128 www-data   16u  IPv6 207472      0t0  TCP fog.myserver.co.uk:http->10.28.10.213:53558 (ESTABLISHED)
    apache2  5520 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5520 www-data   16u  IPv6 209087      0t0  TCP fog.myserver.co.uk:http->10.1.23.231:51926 (ESTABLISHED)
    apache2  5530 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5530 www-data   16u  IPv6 211051      0t0  TCP fog.myserver.co.uk:http->10.240.11.162:64879 (ESTABLISHED)
    apache2  5533 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5533 www-data   16u  IPv6 205433      0t0  TCP fog.myserver.co.uk:http->10.1.37.27:27514 (ESTABLISHED)
    apache2  5544 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5544 www-data   16u  IPv6 205271      0t0  TCP fog.myserver.co.uk:http->10.1.23.148:49740 (ESTABLISHED)
    apache2  5551 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5551 www-data   16u  IPv6 208381      0t0  TCP fog.myserver.co.uk:http->10.9.112.189:49842 (ESTABLISHED)
    apache2  5570 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5570 www-data   16u  IPv6 211113      0t0  TCP fog.myserver.co.uk:http->10.9.106.40:49619 (ESTABLISHED)
    apache2  5572 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5572 www-data   16u  IPv6 208036      0t0  TCP fog.myserver.co.uk:http->10.9.110.128:65140 (ESTABLISHED)
    apache2  5580 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5580 www-data   16u  IPv6 206913      0t0  TCP fog.myserver.co.uk:http->10.1.23.230:49844 (ESTABLISHED)
    apache2  5602 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5602 www-data   16u  IPv6 208805      0t0  TCP fog.myserver.co.uk:http->10.1.24.126:55469 (ESTABLISHED)
    apache2  5618 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5618 www-data   16u  IPv6 206700      0t0  TCP fog.myserver.co.uk:http->10.9.106.29:49608 (ESTABLISHED)
    apache2  5619 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  5619 www-data   16u  IPv6 208974      0t0  TCP fog.myserver.co.uk:http->10.1.55.236:49938 (ESTABLISHED)
    apache2  6006 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6006 www-data   16u  IPv6 209326      0t0  TCP fog.myserver.co.uk:http->10.1.25.118:55158 (ESTABLISHED)
    apache2  6268 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6268 www-data   16u  IPv6 209858      0t0  TCP fog.myserver.co.uk:http->10.1.26.123:53330 (ESTABLISHED)
    apache2  6269 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6269 www-data   16u  IPv6 211026      0t0  TCP fog.myserver.co.uk:http->10.28.12.239:52159 (ESTABLISHED)
    apache2  6270 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6270 www-data   16u  IPv6 211175      0t0  TCP fog.myserver.co.uk:http->10.28.10.93:64841 (ESTABLISHED)
    apache2  6273 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6273 www-data   16u  IPv6 209592      0t0  TCP fog.myserver.co.uk:http->10.9.110.127:58944 (ESTABLISHED)
    apache2  6292 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6292 www-data   16u  IPv6 207455      0t0  TCP fog.myserver.co.uk:http->10.1.21.151:51066 (ESTABLISHED)
    apache2  6296 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6296 www-data   16u  IPv6 209338      0t0  TCP fog.myserver.co.uk:http->10.1.24.160:52020 (ESTABLISHED)
    apache2  6304 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6304 www-data   16u  IPv6 211197      0t0  TCP fog.myserver.co.uk:http->10.9.112.168:54211 (ESTABLISHED)
    apache2  6305 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6305 www-data   16u  IPv6 211145      0t0  TCP fog.myserver.co.uk:http->10.28.11.210:49781 (ESTABLISHED)
    apache2  6787 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6787 www-data   16u  IPv6 209787      0t0  TCP fog.myserver.co.uk:http->pc26-36.myserver.co.uk:49781 (ESTABLISHED)
    apache2  6794 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6794 www-data   16u  IPv6 208319      0t0  TCP fog.myserver.co.uk:http->10.1.23.233:49922 (ESTABLISHED)
    apache2  6796 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2  6796 www-data   16u  IPv6 199442      0t0  TCP fog.myserver.co.uk:http->10.240.11.176:50216 (ESTABLISHED)
    apache2 14670 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14670 www-data   16u  IPv6 208028      0t0  TCP fog.myserver.co.uk:http->10.1.25.150:58961 (ESTABLISHED)
    apache2 14671 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14671 www-data   16u  IPv6 209766      0t0  TCP fog.myserver.co.uk:http->10.28.10.168:60393 (ESTABLISHED)
    apache2 14672 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14672 www-data   16u  IPv6 205388      0t0  TCP fog.myserver.co.uk:http->10.1.23.211:49845 (ESTABLISHED)
    apache2 14673 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14673 www-data   16u  IPv6 211298      0t0  TCP fog.myserver.co.uk:http->10.240.11.236:55911 (ESTABLISHED)
    apache2 14681 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14681 www-data   16u  IPv6 209099      0t0  TCP fog.myserver.co.uk:http->10.1.26.103:51299 (ESTABLISHED)
    apache2 14682 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14682 www-data   16u  IPv6 205304      0t0  TCP fog.myserver.co.uk:http->10.9.112.196:49838 (ESTABLISHED)
    apache2 14684 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14684 www-data   16u  IPv6 211173      0t0  TCP fog.myserver.co.uk:http->10.1.23.209:50288 (ESTABLISHED)
    apache2 14685 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14685 www-data   16u  IPv6 209595      0t0  TCP fog.myserver.co.uk:http->10.1.24.103:49763 (ESTABLISHED)
    apache2 14686 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14686 www-data   16u  IPv6 208799      0t0  TCP fog.myserver.co.uk:http->10.28.10.222:51282 (ESTABLISHED)
    apache2 14687 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14687 www-data   16u  IPv6 208014      0t0  TCP fog.myserver.co.uk:http->10.1.26.194:49692 (ESTABLISHED)
    apache2 14694 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14694 www-data   16u  IPv6 206769      0t0  TCP fog.myserver.co.uk:http->10.1.23.205:51748 (ESTABLISHED)
    apache2 14696 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14696 www-data   16u  IPv6 209771      0t0  TCP fog.myserver.co.uk:http->10.1.26.163:64329 (ESTABLISHED)
    apache2 14697 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14697 www-data   16u  IPv6 207285      0t0  TCP fog.myserver.co.uk:http->10.1.25.174:56202 (ESTABLISHED)
    apache2 14701 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14701 www-data   16u  IPv6 207548      0t0  TCP fog.myserver.co.uk:http->10.28.7.15:63067 (ESTABLISHED)
    apache2 14710 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14710 www-data   16u  IPv6 209927      0t0  TCP fog.myserver.co.uk:http->10.28.10.178:59324 (ESTABLISHED)
    apache2 14713 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14713 www-data   16u  IPv6 209794      0t0  TCP fog.myserver.co.uk:http->10.28.8.20:56839 (ESTABLISHED)
    apache2 14714 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14714 www-data   16u  IPv6 211325      0t0  TCP fog.myserver.co.uk:http->10.28.10.180:49471 (ESTABLISHED)
    apache2 14716 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14716 www-data   16u  IPv6 207539      0t0  TCP fog.myserver.co.uk:http->10.9.106.221:49627 (ESTABLISHED)
    apache2 14725 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14725 www-data   16u  IPv6 209585      0t0  TCP fog.myserver.co.uk:http->pc24-17.myserver.co.uk:65308 (ESTABLISHED)
    apache2 14726 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14726 www-data   16u  IPv6 199604      0t0  TCP fog.myserver.co.uk:http->10.1.25.139:65317 (ESTABLISHED)
    apache2 14727 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14727 www-data   16u  IPv6 215258      0t0  TCP fog.myserver.co.uk:http->10.1.21.170:29597 (ESTABLISHED)
    apache2 14728 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14728 www-data   16u  IPv6 209451      0t0  TCP fog.myserver.co.uk:http->10.1.23.219:51341 (ESTABLISHED)
    apache2 14731 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14731 www-data   16u  IPv6 209752      0t0  TCP fog.myserver.co.uk:http->10.9.106.32:49646 (ESTABLISHED)
    apache2 14734 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14734 www-data   16u  IPv6 206560      0t0  TCP fog.myserver.co.uk:http->10.1.25.143:50171 (ESTABLISHED)
    apache2 14735 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14735 www-data   16u  IPv6 208701      0t0  TCP fog.myserver.co.uk:http->10.9.106.26:50250 (ESTABLISHED)
    apache2 14748 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14748 www-data   16u  IPv6 206878      0t0  TCP fog.myserver.co.uk:http->10.1.25.135:53350 (ESTABLISHED)
    apache2 14756 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14756 www-data   16u  IPv6 205462      0t0  TCP fog.myserver.co.uk:http->10.240.11.232:49580 (ESTABLISHED)
    apache2 14757 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 14757 www-data   16u  IPv6 209305      0t0  TCP fog.myserver.co.uk:http->10.240.11.36:49789 (ESTABLISHED)
    apache2 15211 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15211 www-data   16u  IPv6 205319      0t0  TCP fog.myserver.co.uk:http->10.28.10.194:62332 (ESTABLISHED)
    apache2 15234 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15234 www-data   16u  IPv6 199457      0t0  TCP fog.myserver.co.uk:http->10.9.106.45:50690 (ESTABLISHED)
    apache2 15235 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15235 www-data   16u  IPv6 208384      0t0  TCP fog.myserver.co.uk:http->10.1.26.171:50388 (ESTABLISHED)
    apache2 15282 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15282 www-data   16u  IPv6 205353      0t0  TCP fog.myserver.co.uk:http->10.9.106.31:49660 (ESTABLISHED)
    apache2 15307 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15307 www-data   16u  IPv6 199602      0t0  TCP fog.myserver.co.uk:http->10.28.11.207:52033 (ESTABLISHED)
    apache2 15308 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15308 www-data   16u  IPv6 204923      0t0  TCP fog.myserver.co.uk:http->10.9.112.166:56260 (ESTABLISHED)
    apache2 15352 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15352 www-data   16u  IPv6 211225      0t0  TCP fog.myserver.co.uk:http->10.240.11.199:49886 (ESTABLISHED)
    apache2 15367 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15367 www-data   16u  IPv6 209694      0t0  TCP fog.myserver.co.uk:http->pc26-25.myserver.co.uk:52138 (ESTABLISHED)
    apache2 15370 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15370 www-data   16u  IPv6 209280      0t0  TCP fog.myserver.co.uk:http->10.1.25.156:53894 (ESTABLISHED)
    apache2 15371 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15371 www-data   16u  IPv6 207943      0t0  TCP fog.myserver.co.uk:http->10.240.11.232:49564 (ESTABLISHED)
    apache2 15385 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15385 www-data   16u  IPv6 206583      0t0  TCP fog.myserver.co.uk:http->10.9.106.36:49647 (ESTABLISHED)
    apache2 15389 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15389 www-data   16u  IPv6 211236      0t0  TCP fog.myserver.co.uk:http->10.1.24.160:52021 (ESTABLISHED)
    apache2 15390 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15390 www-data   16u  IPv6 209116      0t0  TCP fog.myserver.co.uk:http->10.1.23.197:50257 (ESTABLISHED)
    apache2 15393 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15393 www-data   16u  IPv6 209989      0t0  TCP fog.myserver.co.uk:http->10.1.26.224:56221 (ESTABLISHED)
    apache2 15396 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15396 www-data   16u  IPv6 209685      0t0  TCP fog.myserver.co.uk:http->10.28.10.24:55542 (ESTABLISHED)
    apache2 15399 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15399 www-data   16u  IPv6 211314      0t0  TCP fog.myserver.co.uk:http->10.1.23.237:56843 (ESTABLISHED)
    apache2 15789 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15789 www-data   16u  IPv6 208833      0t0  TCP fog.myserver.co.uk:http->10.28.7.35:49378 (ESTABLISHED)
    apache2 15790 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15790 www-data   16u  IPv6 207953      0t0  TCP fog.myserver.co.uk:http->10.28.10.95:58542 (ESTABLISHED)
    apache2 15793 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15793 www-data   16u  IPv6 206758      0t0  TCP fog.myserver.co.uk:http->10.1.26.105:50066 (ESTABLISHED)
    apache2 15794 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15794 www-data   16u  IPv6 205274      0t0  TCP fog.myserver.co.uk:http->10.1.23.216:56813 (ESTABLISHED)
    apache2 15795 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15795 www-data   16u  IPv6 206577      0t0  TCP fog.myserver.co.uk:http->10.1.26.94:62086 (ESTABLISHED)
    apache2 15803 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15803 www-data   16u  IPv6 207925      0t0  TCP fog.myserver.co.uk:http->10.28.10.219:64208 (ESTABLISHED)
    apache2 15804 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15804 www-data   16u  IPv6 209654      0t0  TCP fog.myserver.co.uk:http->10.240.11.36:49790 (ESTABLISHED)
    apache2 15806 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15806 www-data   16u  IPv6 205341      0t0  TCP fog.myserver.co.uk:http->10.1.55.226:53041 (ESTABLISHED)
    apache2 15807 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15807 www-data   16u  IPv6 208059      0t0  TCP fog.myserver.co.uk:http->10.9.106.34:49648 (ESTABLISHED)
    apache2 15808 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15808 www-data   16u  IPv6 206831      0t0  TCP fog.myserver.co.uk:http->10.1.26.93:63211 (ESTABLISHED)
    apache2 15809 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15809 www-data   16u  IPv6 209494      0t0  TCP fog.myserver.co.uk:http->10.28.10.203:55371 (FIN_WAIT2)
    apache2 15813 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15813 www-data   16u  IPv6 207512      0t0  TCP fog.myserver.co.uk:http->10.28.10.93:49609 (ESTABLISHED)
    apache2 15814 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15814 www-data   16u  IPv6 209625      0t0  TCP fog.myserver.co.uk:http->10.240.11.226:49533 (ESTABLISHED)
    apache2 15815 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15815 www-data   16u  IPv6 199544      0t0  TCP fog.myserver.co.uk:http->10.1.26.88:49757 (ESTABLISHED)
    apache2 15816 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15816 www-data   16u  IPv6 209879      0t0  TCP fog.myserver.co.uk:http->10.1.55.228:49738 (ESTABLISHED)
    apache2 15817 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15817 www-data   16u  IPv6 208973      0t0  TCP fog.myserver.co.uk:http->10.28.10.168:60392 (ESTABLISHED)
    apache2 15818 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15818 www-data   16u  IPv6 206574      0t0  TCP fog.myserver.co.uk:http->10.28.10.203:55369 (ESTABLISHED)
    apache2 15819 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15819 www-data   16u  IPv6 210971      0t0  TCP fog.myserver.co.uk:http->10.28.12.225:50235 (ESTABLISHED)
    apache2 15820 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15820 www-data   16u  IPv6 208821      0t0  TCP fog.myserver.co.uk:http->10.1.28.51:50042 (ESTABLISHED)
    apache2 15833 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15833 www-data   16u  IPv6 208149      0t0  TCP fog.myserver.co.uk:http->10.28.10.11:54096 (ESTABLISHED)
    apache2 15837 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15837 www-data   16u  IPv6 207433      0t0  TCP fog.myserver.co.uk:http->pc24-23.myserver.co.uk:50093 (ESTABLISHED)
    apache2 15927 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15927 www-data   16u  IPv6 206566      0t0  TCP fog.myserver.co.uk:http->10.1.26.60:50475 (ESTABLISHED)
    apache2 15928 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15928 www-data   16u  IPv6 205446      0t0  TCP fog.myserver.co.uk:http->10.28.10.157:63434 (ESTABLISHED)
    apache2 15929 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 15929 www-data   16u  IPv6 206629      0t0  TCP fog.myserver.co.uk:http->10.28.10.199:54524 (ESTABLISHED)
    apache2 16107 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 16107 www-data   16u  IPv6 208099      0t0  TCP fog.myserver.co.uk:http->10.28.10.14:50199 (ESTABLISHED)
    apache2 17032 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17032 www-data   16u  IPv6 207327      0t0  TCP fog.myserver.co.uk:http->10.28.10.192:51399 (ESTABLISHED)
    apache2 17311 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17311 www-data   16u  IPv6 207968      0t0  TCP fog.myserver.co.uk:http->10.240.11.153:49295 (ESTABLISHED)
    apache2 17315 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17315 www-data   16u  IPv6 208474      0t0  TCP fog.myserver.co.uk:http->10.9.106.27:49633 (ESTABLISHED)
    apache2 17316 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17316 www-data   16u  IPv6 206688      0t0  TCP fog.myserver.co.uk:http->10.9.110.165:56298 (ESTABLISHED)
    apache2 17320 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17320 www-data   16u  IPv6 209074      0t0  TCP fog.myserver.co.uk:http->10.1.24.172:51523 (ESTABLISHED)
    apache2 17321 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17321 www-data   16u  IPv6 208433      0t0  TCP fog.myserver.co.uk:http->10.1.25.151:50002 (ESTABLISHED)
    apache2 17322 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17322 www-data   16u  IPv6 206784      0t0  TCP fog.myserver.co.uk:http->10.28.12.225:50233 (ESTABLISHED)
    apache2 17323 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17323 www-data   16u  IPv6 207321      0t0  TCP fog.myserver.co.uk:http->10.1.25.178:58762 (ESTABLISHED)
    apache2 17328 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17328 www-data   16u  IPv6 206844      0t0  TCP fog.myserver.co.uk:http->10.1.23.239:50334 (ESTABLISHED)
    apache2 17329 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17329 www-data   16u  IPv6 199659      0t0  TCP fog.myserver.co.uk:http->10.240.11.235:49983 (ESTABLISHED)
    apache2 17330 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17330 www-data   16u  IPv6 205477      0t0  TCP fog.myserver.co.uk:http->10.28.10.177:59575 (ESTABLISHED)
    apache2 17335 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17335 www-data   16u  IPv6 208329      0t0  TCP fog.myserver.co.uk:http->10.1.25.129:52700 (ESTABLISHED)
    apache2 17336 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17336 www-data   16u  IPv6 209230      0t0  TCP fog.myserver.co.uk:http->10.1.37.27:27513 (ESTABLISHED)
    apache2 17337 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 17337 www-data   16u  IPv6 199409      0t0  TCP fog.myserver.co.uk:http->10.240.11.182:55925 (ESTABLISHED)
    apache2 18513 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18513 www-data   16u  IPv6 208128      0t0  TCP fog.myserver.co.uk:http->10.1.26.166:63340 (ESTABLISHED)
    apache2 18519 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18519 www-data   16u  IPv6 211274      0t0  TCP fog.myserver.co.uk:http->10.1.25.119:50908 (ESTABLISHED)
    apache2 18525 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18525 www-data   16u  IPv6 207974      0t0  TCP fog.myserver.co.uk:http->10.240.11.199:49885 (ESTABLISHED)
    apache2 18526 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18526 www-data   16u  IPv6 208623      0t0  TCP fog.myserver.co.uk:http->pc24-31.myserver.co.uk:49703 (ESTABLISHED)
    apache2 18530 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18530 www-data   16u  IPv6 209554      0t0  TCP fog.myserver.co.uk:http->10.28.10.155:56252 (ESTABLISHED)
    apache2 18534 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18534 www-data   16u  IPv6 205313      0t0  TCP fog.myserver.co.uk:http->10.28.10.60:52758 (ESTABLISHED)
    apache2 18535 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18535 www-data   16u  IPv6 207876      0t0  TCP fog.myserver.co.uk:http->10.9.106.43:49624 (ESTABLISHED)
    apache2 18542 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18542 www-data   16u  IPv6 211385      0t0  TCP fog.myserver.co.uk:http->10.28.11.223:50357 (ESTABLISHED)
    apache2 18555 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18555 www-data   16u  IPv6 206477      0t0  TCP fog.myserver.co.uk:http->10.1.55.107:51872 (FIN_WAIT2)
    apache2 18560 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18560 www-data   16u  IPv6 206707      0t0  TCP fog.myserver.co.uk:http->10.28.12.239:52158 (ESTABLISHED)
    apache2 18561 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18561 www-data   16u  IPv6 209474      0t0  TCP fog.myserver.co.uk:http->10.1.26.166:63341 (FIN_WAIT2)
    apache2 18564 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18564 www-data   16u  IPv6 206586      0t0  TCP fog.myserver.co.uk:http->10.28.7.60:49773 (ESTABLISHED)
    apache2 18565 www-data    4u  IPv6  12460      0t0  TCP *:http (LISTEN)
    apache2 18565 www-data   16u  IPv6 208156      0t0  TCP fog.myserver.co.uk:http->10.28.10.92:58723 (ESTABLISHED)
    root@fog:/var/log/mysql#
    
    

    Will keep an eye on what people do :D


  • Developer

    What does lsof -i :3306 show? Is it all localhost connections?

    And what about lsof -i :80? All connections from different clients??



  • Right now i have 33525 mysql connections when i run “show status like ‘Con%’;” in mysql cli. And no…that was not a mistype :D The connections are just not going away.

    Total registered hosts are 1892 of which about 1800 have the fog client.

    Running lsof, I am a around 20-30 established connections at any one given time with about the same listening. All while the CPU is around 100%. Every once and a while it might spike to around 180 or so but then drop back down within 30 seconds or so.

    FOG_HOST_LOOKUP was checked and I just unchecked it to see if that helps. So far I am not seeing a difference after a total system reboot.

    I had to restart the mysqld process so that I could get to that setting as the web gui would not do anything. In that 2 minutes from a reboot, I am back upto 297 mysql connections.

    BTW, I am running this on RedHat Enterprise 6.

    Thanks!

    Adam


  • Developer

    @Adam-Taylor Please install the packet lsof on your FOG server and run lsof -i :80 to see who keeps all those apache processes busy.


  • Senior Developer

    @Adam-Taylor Do you have a lot of clients? Are you using FOG_HOST_LOOKUP? (On hosts do you see the green/red dots when on list/search?)


  • Senior Developer

    @Adam-Taylor No, nothing has been changed on the vhost generation. Maybe you have a lot more computers running and creating new connections?



  • And i forgot.

    The message only happens every once and a while in the error_log file.



  • Rebooted…1 hour later still seeing same thing. I’m thinking there is something going on with apache. While i write this there is 259 httpd processes chewing up everything.

    In the apache log, i get:

    [Thu Oct 22 16:54:22 2015] [error] server reached MaxClients setting, consider raising the MaxClients setting

    I just don’t know what would be causing this all of a sudden… Was something changed in the apache v/host settings in the past few weeks with the installer?

    Adam


  • Senior Developer

    Actually I’d recommend doing a full server reboot. I think what you’re seeing now is it catching up from all the bad stuff before. Of course it may be the case that something else is weird. Can you watch your Apache error log? Often when I see delays of loading as such there is usually something at a fast and large pace writing to the log files.



  • Sorry, I haven’t gotten back lately…had a sick child.

    I have tried 5040 and 5042 this morning and I am still seeing almost complete CPU usage to the point that the web gui will talk 20-30 seconds to respond or not respond at all or not render most of the page.

    I will try to go back to older ones to see where the problem was introduced.


  • Senior Developer

    @Matthieu-Jacquart et al, I believe this was fixed last night. Something didn’t seem right and while chatting what I thought the issue was it came to me. Thank you @Uncle-Frank and everybody for reporting and patience. Basically my suspicion of what was happening is the gui kept on make connections to the db. Anytime a new page was loaded or clients where checking in a new db connection was being created even if you were using persistent connections. I looked over the code and it certainly made sense based on the logic of the code. I should’ve thought of it sooner I know, but I’m glad it’s now working more appropriately.



  • It’s seems ok with last svn (git 5040)
    Load Average 1.16, 2.65, 2.48



  • @Tom-Elliott For me too, cpu is running 100% due to apache2 and mysqld processes after svn 5020 upgrade.
    In web interface I’ve got these values : Load Average 96.24, 101.58, 106.29
    Not the first time I’ve got this issue after upgrading svn ;)



  • I forgot to add.

    We have 1892 hosts currently.

    As for the logs…the only logs i am getting are:

    [Tue Oct 20 16:09:34 2015] [error] [client 10.69.15.60] PHP Warning: mysqli::real_connect(): (08004/1040): Too many connections in /var/www/html/fog/lib/db/MySQL.class.php on line 29

    This is when the client hits the server to see if it has any tasks.

    If i set the max_connections in sql to something huge (2k plus), in about 45 minutes i will start to get that message. The connections just keep racking up and up…never go down…

    I also tried setting “wait_timeout” and “interactive_timeout” to something like 10 seconds to try and make the connections go down…nada…connections keep rising.

    I’m about to get off work, but first thing in the morning i am going to start bouncing around the svn versions to see if i can find which one might be causing this.

    Thanks!

    Adam



  • @Adam-Taylor You can export the DB before each change.

    FOG Configuration -> Configuration Save -> Export

    That exports the entire DB. Everything. I notate my exports name with a revision number post-download.

    So all my exports are named “filename_rxxxx_month_year”


  • Senior Developer

    While it is never a good idea to revert, none of the recent db changes are going to affect anything. If anything else, they will simply be ignored in revisions where they were not in use. Where you would run into problems is if you try to backtrack say all the way back to version 1.2.0 on your current database. This is because I removed necessary bits that 1.2 expects, but is not necessary in the current. Just don’t downgrade directly to one such revision and you will be fine.



  • I understand what you are saying and i am happy to help try and figure out where it went south. My only question is the DB schema backtracking…

    Do you know how far i can go back in revisions before i hit a change that will cause a problem? I know fog can’t downgrade so i want to make sure i don’t blow up the DB :-)

    Adam



  • @Tom-Elliott @Adam-Taylor

    Tom is suggesting you figure out where it went wrong… somewhere between 4960 and 5020 obviously.

    You can checkout a specific revision like this:

    svn -r xxxx up

    or

    svn -r 4174 up

    Of course, install it afterwards: ./installfog.sh -y

    It really helps a lot if FOG Trunk is virtualized… because again I’d stress fog does not support downgrading… but if Tom says it’s OK to do for this small range of SVN revisions, then it’s OK to do.



  • Apologies.

    The @Developers are working on it. FOG has recently moved to innodb, and with that comes many challenges and necessary tweaks.

    SVN version 4103 is pretty solid. But keep in mind FOG does not support downgrading.

    Please provide any relevant logs. How many hosts do you have in your environment?


Log in to reply
 

385
Online

6.6k
Users

14.0k
Topics

132.3k
Posts