Issues with Fog 1.5.2

[K.Hays]

@sebastian-roth Yessir!

[K.Hays]

Welpppp, I suppose not. We were running this server on a hyper v and decided to just try out to separate options. We had a new computer we were going to try and load it on, as well as setting up a new hyper v with debian instead. Both fresh installs had the same error when we tried to test either of them. I went ahead and checked the file the George mentioned prior ( On the Hyper V server) and this is the outcome.

[K.Hays]

@george1421 I adjusted the file to match what you put in this post https://forums.fogproject.org/topic/11797/updated-from-fog-v1-50-to-v1-52-issue/25

Will test soon. Any other ideas?

[Quazz]

@k-hays Can you deploy an image to a single client without issue?

This seems to me like an issue with PHP-FPM getting overloaded.

On debian the pm.max_children directive is often stuck on 5 which is generally too low for most people. (40 is a good starting point for testing)

[K.Hays]

@quazz Yes. Single client works fine. I don’t know the specific number in which it starts to fail; but in a lab we image anywhere from 20 - 35ish computers and it seems to fail whenever we try those numbers. The issue also persisted when we started with ubuntu.

[Quazz]

@k-hays There’s currently some known issues with PHP-FPM settings/getting overloaded, particularily on debian based systems.

I definitely recommend checking the pm.max_children value (not 100% sure where it is on debian)

You should be able to check the PHP-FPM logs on the WebUI to see if it’s been complaining about not enough max_children or memory exhaustion or timeouts I think.

[K.Hays]

@quazz I’ll go digging. Any ideas?

[Quazz]

@k-hays Try grep -irl pm.max_children /etc

[K.Hays]

@quazz Ok I found it! Now, what do you think the max i should put there is. Sometimes we might image up to two, maybe three labs at a time. would 90 be a stretch?

[Quazz]

@k-hays It depends on how many resources your server has available to itself.

We haven’t done a ton of testing on the exact numbers. The value will depend on the amount of RAM and how much the average PHP-FPM uses.

Start with 40 as a safe value and go from there, imo.

[K.Hays]

@quazz Ok will do. I’ll end up testing this today as well. We’re using a server that has a lot of unused resources so we can devote a good bit to it. I also want to say that we are on the current dev-branch, so 1.5.4. These were all fresh builds as well, even on ubuntu. Does it automatically set the max children to 5 now?

[george1421]

@k-hays said in Issues with Fog 1.5.2:

. would 90 be a stretch?

Don’t do that much you could run into a situation of resource exhaustion. Under normal imaging you shouldn’t see more than 10 worker thread. For max children I would set it to 40.

If you are using FOG 1.5.2 or later here is what I would change in the www.conf file for php-fpm.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 40
pm.min_spare_servers = 6
pm.start_servers = 5

Update those settings then restart php-fpm.

Queue up your multicast then on the fog server linux console start top then press P to sort by CPU usage. Watch, you should have 5-7 php-fpm worker threads running, they should be the top cpu users.

[K.Hays]

@george1421 Ok, does it make a difference if we do unicast? Just wondering

[george1421]

@k-hays The memory setting will help the unicast imaging too

But just to be sure none of your conditions have changed since your original post of 2 months ago? You are still having the same exact condition?

[K.Hays]

In terms of network yes. The server itself is the same, but it’s been reloaded to debian 9 and 1.5.4 as opposed to ubuntu and 1.5.2.

[K.Hays]

@k-hays Also if you meant the error, then yes. It is the exact same issue (or at least extremely similar). I made the changes and will test it again shortly.

[K.Hays]

So PHP wouldn’t restart and i’m pretty sure it was because i had set my min spare servers larger than the max. That being said, if my min is 6 (as you said) what should my max be?

Edit- @george1421 I set it to 6 just to see and PHP still failed with a different error this time. It says the start server can be less then the min spare and no more than the max. So what should i change them to?

[george1421]

@k-hays Max should be 40 as I posted below.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 40
pm.min_spare_servers = 6
pm.start_servers = 5

[K.Hays]

@george1421 Yes, and it is currently set to 40. When i said max i meant the Max_spare_servers option. So what it’s telling me is that pm.start_servers cannot be a smaller number than pm.min_spare_servers. Or at least thats the error i’m getting. The last things you had me change (pm.Start_servers and pm,.mid_spare_servers) are apparently causing the error. Also the pm.max_spare_servers that needs to be set higher than what we set the start_servers options as well. could i just set them to 5,6,7 respectively? Im sorry, i’m really not entirely sure how this works

[K.Hays]

@george1421 Another update. I set Min_spare_servers = 6, max_spare_servers = 6, and start_servers = 6 and php successfully restarted/not sure if thats good or bad.