WIN10 Multicast Imaging Issues

[Joe Gill]

@george1421

I believe I have already done this.

[george1421]

@joe-gill Well then its pretty clear that php-fpm isn’t causing the multicasting issue. (nuts). Nothing of any value shows up in the multicasting logs when things go sideways?

[Joe Gill]

@george1421
I can post any logs, if you’d like. Multicast does complete. The image just does not function. I was out sick the last couple days. I’m back at it today. I’m hoping to get my new image done here this week. We’ll see.

[Joe Gill]

@george1421

The multicast logs look normal.

[george1421]

@joe-gill So is imaging working correctly now with apache-php? If so I want to break it to find out what is going sideways.

[Joe Gill]

@george1421
Multicast fails on both Apache-PHP and FPM.

[Tom Elliott]

@joe-gill mind installing working branch to see if it helps with mulyicast? I’d also recommend trying the rendezvous option for mulyicast. This can be setup in fog settings. Set the address to the ip of your fog server, then as a failsafe restart the multicast manager service and then the clients.

[Joe Gill]

@tom-elliott
Thanks! Will do! I’ll report back. I will try a few labs today and see what happens.

[Joe Gill]

@tom-elliott
@george1421

So… I tasked 5 machines for a test batch. Multicast worked on those 5 just fine. The image seems to be running fine. The task session was extremely fast. They were finished in less than 8 minutes.

I tasked 30 more… and the interface locks up. I cannot get to the interface now at all.

My server is running Debian 9.4. It ran fine using Apache. Now that we are back to FPM, the interface is locking up tight.

Thanks guys! I’ll be around today until noon MST than again tomorrow 7 - 3 MST.

[Joe Gill]

I restarted PHP-FPM and the interface came back up.

[Joe Gill]

Well I think I may have something for you guys now.

I had a batch of clones multitasking after I restarted the PHP-FPM services when things locked up… Some of the clones in the group did not start. Some did. The ones that started hung at this screen.

Than after a bit of time, I noticed the interface was locked up once again… So I restarted PHP-FPM services and got the following message…

Let me know if you need any log files. Thanks!

[Joe Gill]

@Tom-Elliott
@george1421

This might help with the Debian 9 issue.

Looks like the pm.max_childeren setting has been reached. That is when the server interface locks up.

[Tom Elliott]

@joe-gill can you try setting max children to say 200 and restart php fpm

[Joe Gill]

@tom-elliott
I am playing with it right now. Based on having 4 gb of RAM allotted for the server. I came up with these from another website about php-fpm.

I reset the settings to the following:

pm.max_children = 40
pm.start_servers = 15
pm.min_spare_servers = 15
pm.max_spare_servers = 25
pm.max_requests = 500

So far PHP-FPM hasn’t bombed out. I will report back after I run a few more things.

Thanks!

[george1421]

@joe-gill Excellent find.

OK we can start debugging (i.e. better understanding what is going on) from here.

When you are running your multicast with 20-30 systems. I need you to run a command on the fog server.
ps -ylC php-fpm --sort:rss
Will print out all of the php-fpm processes. We are interested in the RSS column. In you mind, just give an estimate of the average size of the memory consumed.

FWIW: ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"Mb") }' will average the memory consumed by all worker threads.

Once imaging is done stop php-fpm and check the free ram and record that value.

Report back the number.

The next test is I want to see if there is a direct correlation between the number of php-fpm worker threads and the number of hosts in your multicast pool. With your new numbers you will need more than 15 hosts in your deployment queue to see any change in the number of worker threads.

You can probably drop back the start_servers, min_spare_servers back to 5, and then drop your max_spare_servers back to 8-10.

The idea is the worker threads are dynamic. Starting up with 5 is not a bad number, and setting min spare will leave 5 running even if there is not requests coming in.

There is a default timer if a worker thread is not awakened in 10 seconds the php-fpm management process will kill it off returning its memory back to the system. The management process will kill off stagnant workers sitting around for 10 seconds until the min_spare_servers is reached. In your case if there are any worker counts above 25, they will be killed immediately to get back to 25 spare threads. The point of all of this is a dance between workers sitting around not doing anything but consuming memory and having system resources.

My suspicion is that there is a direct correlation between multicast clients and worker threads (1:1). Setting the number of max_children too high might starve your FOG server of ram.

[Joe Gill]

@george1421
Very interesting! I will run those tests here later this morning. The images so far have worked with Multicasting also. I haven’t had any issues with loosing interactivity with my FOG interface either since I adjusted things.

Stay tuned. I’ll have some results to post here in about an hour or so.

[Joe Gill]

@george1421

OK. So here is some feedback.

I ran ps -ylC php-fpm --sort:rss with 27 clients running a multicast session. I did not get any results at all returned.

I did have php-fpm7.0 -F -c /etc/php/7.0/fpm/php-fpm.conf running. In fact it had been running since I ran my last Multicast sessions. I hadn’t received any more errors until this morning when I ran a batch. Incidentally, when I received the warning below, I ran the command above and it still didn’t return anything.

I have that multicast session open as we speak. So if their are any other things you’d like me to try PM me.

[george1421]

@joe-gill Do you or can you call up the info.php page you created. We need to make sure that apache is now using php-fpm. Your picture from many posts ago show apache php is currently configured.

[Joe Gill]

@george1421

I killed the php-fpm7.0 -F -c /etc/php/7.0/fpm/php-fpm.conf and had not restarted it. I just re ran that command and PHP-FPM is now running again. Still no results from /etc/php/7.0/fpm/pool.d# ps -ylC php-fpm --sort:rss.

Should I restart that multicast session and try this all over again? I certainly can.

[george1421]

I remoted into Joe’s system and looked about for a bit.

What I found was there was/is 3 versions of php and php-fpm installed and all were trying to run at the same time.

I stopped, php-fpm v5, php-fpm v7.0, and left php-fpm 7.2 running.

I could not find a handy copy of the official fog configuration for the www.conf file so I hand edited the distro’s version setting max processes to 50, start, min to 5 and spares to 6. I also set php max memory size to 256M.

After that I asked Joe to multicast deploy an image. He reported that he deployed to 24 systems. I watched the process status screen and the number of active php-fpm processes never went above 6 workers (where 50 is the max). Joe stated that all 24 machines deployed to completion.

So at this point I’m pretty sure the issue with this system was having multiple versions of php installed as the root cause. I’m not ready to rule out a www.conf file setting that is not right, but I’m happy with what I saw from the performance of php-fpm service in this multicast deployment.