WIN10 Multicast Imaging Issues



  • I just made the switch over to WIN10 in our district. I just got finished making my sysprepped images for WIN10.

    I followed: https://forums.fogproject.org/topic/9877/windows-10-pro-oem-sysprep-imaging

    I was able to successfully push my image to a lab full of computers. Everything looked to be going so well. I tried pushing my image to another lab full of identical computers (minus SSD drives) and received the following message on boot up…

    0_1528475578139_IMG_20180608_101242.jpg

    I tried pressing Enter or F8 and it fails with this same message…

    I tried re-doing my image on one of the lab computers with the SSD drive and that failed with the same message. I can push this just fine via Unicast but it will not Multicast.

    Any suggestions here?

    Let me know which log files to upload and I will gladly post them. Thanks!


  • Moderator

    I remoted into Joe’s system and looked about for a bit.

    What I found was there was/is 3 versions of php and php-fpm installed and all were trying to run at the same time.

    I stopped, php-fpm v5, php-fpm v7.0, and left php-fpm 7.2 running.

    I could not find a handy copy of the official fog configuration for the www.conf file so I hand edited the distro’s version setting max processes to 50, start, min to 5 and spares to 6. I also set php max memory size to 256M.

    After that I asked Joe to multicast deploy an image. He reported that he deployed to 24 systems. I watched the process status screen and the number of active php-fpm processes never went above 6 workers (where 50 is the max). Joe stated that all 24 machines deployed to completion.

    So at this point I’m pretty sure the issue with this system was having multiple versions of php installed as the root cause. I’m not ready to rule out a www.conf file setting that is not right, but I’m happy with what I saw from the performance of php-fpm service in this multicast deployment.



  • @george1421

    I killed the php-fpm7.0 -F -c /etc/php/7.0/fpm/php-fpm.conf and had not restarted it. I just re ran that command and PHP-FPM is now running again. Still no results from /etc/php/7.0/fpm/pool.d# ps -ylC php-fpm --sort:rss.

    Should I restart that multicast session and try this all over again? I certainly can.


  • Moderator

    @joe-gill Do you or can you call up the info.php page you created. We need to make sure that apache is now using php-fpm. Your picture from many posts ago show apache php is currently configured.



  • @george1421

    OK. So here is some feedback.

    I ran ps -ylC php-fpm --sort:rss with 27 clients running a multicast session. I did not get any results at all returned.

    0_1529082844498_f0e39a2c-8602-4372-b0aa-26a06f32efa5-image.png

    I did have php-fpm7.0 -F -c /etc/php/7.0/fpm/php-fpm.conf running. In fact it had been running since I ran my last Multicast sessions. I hadn’t received any more errors until this morning when I ran a batch. Incidentally, when I received the warning below, I ran the command above and it still didn’t return anything.

    0_1529082781337_ec95aeb0-fdcb-4385-993c-a18890bd6836-image.png

    I have that multicast session open as we speak. So if their are any other things you’d like me to try PM me.



  • @george1421
    Very interesting! I will run those tests here later this morning. The images so far have worked with Multicasting also. I haven’t had any issues with loosing interactivity with my FOG interface either since I adjusted things.

    Stay tuned. I’ll have some results to post here in about an hour or so.


  • Moderator

    @joe-gill Excellent find.

    OK we can start debugging (i.e. better understanding what is going on) from here.

    When you are running your multicast with 20-30 systems. I need you to run a command on the fog server.
    ps -ylC php-fpm --sort:rss
    Will print out all of the php-fpm processes. We are interested in the RSS column. In you mind, just give an estimate of the average size of the memory consumed.

    FWIW: ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"Mb") }' will average the memory consumed by all worker threads.

    Once imaging is done stop php-fpm and check the free ram and record that value.

    Report back the number.

    The next test is I want to see if there is a direct correlation between the number of php-fpm worker threads and the number of hosts in your multicast pool. With your new numbers you will need more than 15 hosts in your deployment queue to see any change in the number of worker threads.

    You can probably drop back the start_servers, min_spare_servers back to 5, and then drop your max_spare_servers back to 8-10.

    The idea is the worker threads are dynamic. Starting up with 5 is not a bad number, and setting min spare will leave 5 running even if there is not requests coming in.

    There is a default timer if a worker thread is not awakened in 10 seconds the php-fpm management process will kill it off returning its memory back to the system. The management process will kill off stagnant workers sitting around for 10 seconds until the min_spare_servers is reached. In your case if there are any worker counts above 25, they will be killed immediately to get back to 25 spare threads. The point of all of this is a dance between workers sitting around not doing anything but consuming memory and having system resources.

    My suspicion is that there is a direct correlation between multicast clients and worker threads (1:1). Setting the number of max_children too high might starve your FOG server of ram.



  • @tom-elliott
    I am playing with it right now. Based on having 4 gb of RAM allotted for the server. I came up with these from another website about php-fpm.

    I reset the settings to the following:

    pm.max_children = 40
    pm.start_servers = 15
    pm.min_spare_servers = 15
    pm.max_spare_servers = 25
    pm.max_requests = 500

    So far PHP-FPM hasn’t bombed out. I will report back after I run a few more things.

    Thanks!


  • Senior Developer

    @joe-gill can you try setting max children to say 200 and restart php fpm



  • @Tom-Elliott
    @george1421

    This might help with the Debian 9 issue.

    0_1528994080142_Screenshot from 2018-06-14 10-33-30.png

    Looks like the pm.max_childeren setting has been reached. That is when the server interface locks up.



  • Well I think I may have something for you guys now.

    I had a batch of clones multitasking after I restarted the PHP-FPM services when things locked up… Some of the clones in the group did not start. Some did. The ones that started hung at this screen.

    0_1528990027796_IMG_20180614_091812.jpg

    Than after a bit of time, I noticed the interface was locked up once again… So I restarted PHP-FPM services and got the following message…

    0_1528990162953_IMG_20180614_092017.jpg

    Let me know if you need any log files. Thanks!



  • I restarted PHP-FPM and the interface came back up.



  • @tom-elliott
    @george1421

    So… I tasked 5 machines for a test batch. Multicast worked on those 5 just fine. The image seems to be running fine. The task session was extremely fast. They were finished in less than 8 minutes.

    I tasked 30 more… and the interface locks up. I cannot get to the interface now at all.

    My server is running Debian 9.4. It ran fine using Apache. Now that we are back to FPM, the interface is locking up tight.

    Thanks guys! I’ll be around today until noon MST than again tomorrow 7 - 3 MST.



  • @tom-elliott
    Thanks! Will do! I’ll report back. I will try a few labs today and see what happens.


  • Senior Developer

    @joe-gill mind installing working branch to see if it helps with mulyicast? I’d also recommend trying the rendezvous option for mulyicast. This can be setup in fog settings. Set the address to the ip of your fog server, then as a failsafe restart the multicast manager service and then the clients.



  • @george1421
    Multicast fails on both Apache-PHP and FPM.


  • Moderator

    @joe-gill So is imaging working correctly now with apache-php? If so I want to break it to find out what is going sideways.



  • @george1421

    The multicast logs look normal.



  • @george1421
    I can post any logs, if you’d like. Multicast does complete. The image just does not function. I was out sick the last couple days. I’m back at it today. I’m hoping to get my new image done here this week. We’ll see.


  • Moderator

    @joe-gill Well then its pretty clear that php-fpm isn’t causing the multicasting issue. (nuts). Nothing of any value shows up in the multicasting logs when things go sideways?


 

353
Online

41.8k
Users

12.3k
Topics

116.0k
Posts