Solved Issues with Fog 1.5.2
-
OS: Ubuntu 16.04
Fog Version: 1.5.2/ 1.5.0Recently, within the past month or so, we rebuilt our fog machine on a Hyper V server. We started mostly from scratch aside from CSV’s for the groups. We originally installed 1.5.0 and then updated when 1.5.2 came out. It all seemed to work well until last week. The first sign of trouble was when we tried to go to the web gui, It came up with an error along the lines of “Could not connect to database”. I figured this was caused from the issue stemming from Ubuntu. I ran the fix shown on the fog forums, and the gui came back up. The day after that we decided to re image a lab that was having some issues. It was the first time we’d tried a large deployment with the new server. We created the image fine, but when unicasting it to thirty computers 10, 5, and 2 at a time (we tried all of those to see if somehow it was just getting bogged down) the web gui would freeze and crash. The ones prompted to start would start, but the ones in queue would fall out of queue and show the error in the pictures below. We did a bit of experimenting but couldn’t figure it out. We had to get the lab up and running quick though so we went back to a checkpoint before we updated to Fog 1.5.2 and it seemed to work without issue. It is still working on 1.5.0, but I’m wondering why it bogged out when running 1.5.2? I can create another fog server on the hyper v and install 1.5.2 for testing if need be.
-
@george1421 Gotchya, that explains it, Thank you!
-
@k-hays Between FOG 1.5.0 and 1.5.1 (or 1.5.1 and 1.5.2 sorry I can’t remember) the developers switched from using the built in php engine in apache to using a dedicated php engine (php-fpm). This was done to address an issue with GUI slowness reported with the new 1.5.x Web GUI on certain distributions. Baseline testing then showed an excellent improvement in GUI responsiveness as well as client check ins were not causing an issue with heavy web server load as they were before.
After 1.5.2 was released the developers started seeing other reports of issues with multicasting and unicasting in larger environment that wasn’t see during development. So certain tweaks to the php-fpm config files were made in 1.5.3 and 1.5.4. Increasing the memory limit even more to 256MB resolved a multicasting issue was discovered after 1.5.4. was released. That setting will be in 1.5.5 when its released in a few months. With these new settings you should see a pretty stable multicasting deployment.
-
@george1421 Also, did these settings change after a certain update? On our old server we had these issues after an update when we were previously getting by alright.
-
@george1421 So far so good. We were able to deploy to an entire lab using unicast. No errors yet! I’ll be using it more this week and will provide further feedback. Thank you!!
-
@k-hays Not for the worker threads. We are finding (so far) 256MB per php-fpm worker is working well. Run with these setting and for sure provide feedback. Once we get these number working stable on all that are having issues we will include them in the fog 1.5.5 update.
-
@george1421 Yep the memory has been changed since this morning; and i will change the max server when the tasks complete. Thank you !! can or should the memory go higher? The server has 32 gigs of ram, not that it needs it.
-
@k-hays Please, only change MAX server AND make sure you added the memory line for 256M. The rest of the values are ok at 6.
-
Well we went ahead and tested it the the _servers options set to 6,6,6 respectively just to see. It is in fact working. The image is running right now but we waited for the next batch in the unicast task to start and they did. We hadn’t previously gotten the first batch to finish successfully before that so it’s lookin good. So the only other thing you think i should change then would be max_spare_servers to 40; not either of the other min_servers/startserver settings from 6?
-
@k-hays also we have been able to get to the web gui recently without issue.
-
@george1421 ok so max_spare_servers needs to be 40. gotchya. Should i change the other 2 _servers settings from 6? Also should max_spare_servers be set to the same as number as pm.max_children?
-
@k-hays said in Issues with Fog 1.5.2:
max_spare_servers = 6
Set that value to 40 and make sure you have the memory allocated to 256M and you should be good to go. Understand this fix will only address imaging and not your initial post of html in the FOS screen.
-
@george1421 Another update. I set Min_spare_servers = 6, max_spare_servers = 6, and start_servers = 6 and php successfully restarted/not sure if thats good or bad.
-
@george1421 Yes, and it is currently set to 40. When i said max i meant the Max_spare_servers option. So what it’s telling me is that pm.start_servers cannot be a smaller number than pm.min_spare_servers. Or at least thats the error i’m getting. The last things you had me change (pm.Start_servers and pm,.mid_spare_servers) are apparently causing the error. Also the pm.max_spare_servers that needs to be set higher than what we set the start_servers options as well. could i just set them to 5,6,7 respectively? Im sorry, i’m really not entirely sure how this works
-
@k-hays Max should be 40 as I posted below.
php_admin_value[memory_limit] = 256M pm.max_requests = 2000 pm.max_children = 40 pm.min_spare_servers = 6 pm.start_servers = 5 -
So PHP wouldn’t restart and i’m pretty sure it was because i had set my min spare servers larger than the max. That being said, if my min is 6 (as you said) what should my max be?
Edit- @george1421 I set it to 6 just to see and PHP still failed with a different error this time. It says the start server can be less then the min spare and no more than the max. So what should i change them to?
-
@k-hays Also if you meant the error, then yes. It is the exact same issue (or at least extremely similar). I made the changes and will test it again shortly.
-
In terms of network yes. The server itself is the same, but it’s been reloaded to debian 9 and 1.5.4 as opposed to ubuntu and 1.5.2.
-
@k-hays The memory setting will help the unicast imaging too
But just to be sure none of your conditions have changed since your original post of 2 months ago? You are still having the same exact condition?
-
@george1421 Ok, does it make a difference if we do unicast? Just wondering
