503 Error and stuck at "attempting to update database"
-
@rodluz What path did you find it in? There should (really) only be one www.conf in the php path.
If it was commented out the default may be 32MB. Set it to 256M save the file and reboot. That should address the failure under heavy load, basically it was running out of process memory and couldn’t recycle fast enough.
-
@george1421 I found it in /etc/php/7.1/fpm/pool.d
-
@rodluz ok that is the right one, it shifts around depending on the distro and the version of php installed. Adjust the values as I posted then reboot the fog server. Oh, your fog server should have at least 4GB of ram allocated to it. It only need 1 or 2 vCPU. You may need more vCPU if you have a large campus where more than 500 computers running the FOG Client will be hitting your FOG server.
-
@george1421 Thank you so much! I just did that and will test it by imaging more labs. If anything I keep having this issue today or tomorrow I will update the thread.
-
@george1421 Made the changes on the server and deployed the image to a lab of 20 computers and it’s giving me the same issue.
-
@rodluz OK we’ll need to look in /var/log/php-fpm directory, there should be an error log in there. We’ll need to see what is throwing the error. If there isn’t anything usable in there, then check the apache error log. But does sound like a memory exhaustion issue.
So just to be clear you are doing 20 simultaneous unicast images and its throwing the 503 error? (personally that’s a bit more than a single 1 GbE link can manage, but this error is that php-fpm is not responding back to apache within the apache timeout value.
-
@george1421 I checked the logs and I think I found the issue. I’m not at my office anymore so I’ll test it tomorrow and update the thread.
Thank you so much for your support, quick response, and excellent help!! -
@rodluz said in 503 Error and stuck at "attempting to update database":
Made the changes on the server and deployed the image to a lab of 20 computers and it’s giving me the same issue.
Did you actually restart service php7.1-fpm or the whole server to make it apply those changes?
-
@Sebastian-Roth I restarted the service and the server too just to be sure.
-
@george1421 So I am pretty sure it is fixed now. I looked at the logs for php and found this error
[20-Aug-2019 17:28:19] WARNING: [pool www] server reached pm.max_children setting (5), consider raising itSo I changed the pm.max_children setting to 150, just to be sure, and it is now working perfectly!
I re-imaged the same lab with 20 computers and monitored the server to see how many php-fpm services ran and I found that at one point I had 10 instances so bringing that up from 5 did help for sure. -
@rodluz said in 503 Error and stuck at "attempting to update database":
So I changed the pm.max_children setting to 150
Good catch on the max_children setting. If you would have had a current version of FOG that setting would be 35. I think 150 is a bit excessive since each child consumes ram. I think 50 might be a better choice to start with. Most people don’t start 20 simultaneous unicast streams so I can say we’ve not run into this issue before with the standard 35 max children. 5 IS a bit low.
So to recap you had 2 issues, one was the memory limit and the other was the max children setting.
Once you get past your image push crunch, you might consider upgrading to the latest version of FOG.
