Rolling Reboot -- FOG Client AD issue



  • @Developers
    @Joe-Schmitt

    Wed Jul 27, 2016 12:55 pm
    Running Version 1.3.0-RC-4
    SVN Revision: 5941

    I am having a very strange issue with my Hosts joining the domain. I am now using a sysprepped image and am updating AD. I have created a SetupComplete script that runs the FOG service re-activation prior to installing AV and prior to driver injection as my current image is a Windows 7 Universal image.

    Currently the issue I’ve run into is that some machines get stuck in a reboot cycle and that particular machine’s active task never finishes in FOG. The machine physically finishes but the task never ends. It appears that the FOG client is the culprit for restarting the machine.

    The host name on the machine changes (sometimes) but it never adds itself to AD. When you try and add it manually, it gives you the “Welcome to the Domain” message but it never finalizes the add process. In other words it reboots, than it goes back into reboot hell.

    This happens on about 1 out of 5 machines in a multi task group.

    A side issue that is somewhat related is that most hosts add successfully, but some will rename correctly but fail to join AD. About 1 in 5 do this as well.

    TIA!

    Cheers!

    Joe Gill


  • Moderator

    OP had AntiVirus installing inside SetupComplete.cmd before the FOG Client was re-enabled and started. The specific AntiVirus installation caused a reboot, which left the SetupComplete.cmd un-finished, and therefore all these other problems happened.

    Consider building a snapin for the AntiVirus, which will auto-queue for any computer that is told to image and has the snapin associated with it, and the snapin will deploy after imaging is completed.



  • @Wayne-Workman
    @Tom-Elliott
    @Developers

    The new image seems to work flawlessly, like it should! I’ll have to adjust my script on the other image and go from there. I’m glad I finally managed to figure this one out.

    You can change this one to solved! :-)

    Cheers!


  • Moderator

    @Joe-Gill said in Rolling Reboot -- FOG Client AD issue:

    What I failed to realize is that this causes the machine to reboot before the script finishes and it does not let FOG finish.

    Bingo.

    If it wasn’t your setup, we’d be having a dozen new threads every day about the same issue.



  • @Tom-Elliott
    @Developers

    Well I am currently running version RC-8. I am still having the rolling reboot issue with one of my images. I am deploying a new image this morning on a few remaining machines. I’m wondering if it’s not my image instead of FOG. In the image I’m deploying this morning I have changed the order in which the SetupComplete finishes. I was having SetupComplete install AV first and then do some other tasks… What I failed to realize is that this causes the machine to reboot before the script finishes and it does not let FOG finish. I had a similar issue with snapins not finishing because of a reboot. If the new image seems to resolve this, I’ll be sure to let you all know.

    Thanks for all of the support!

    Cheers,

    Joe Gill



  • @Tom-Elliott

    Well after an update, I still had 1 out of 4 clients do the rolling reboot. It’s interesting because the one that was stuck, was a different model. I"m wondering if their isn’t a driver issue with the image as it’s a universal image. The only funny thing is that, again, after you delete the task from the active task list, things proceed as they should. AD add works, drivers seem fine, everything is happy. So I’m still a little stumped. LOL!

    If you’d like any logs just say so and I’ll post whatever you need.

    I’ll be away from the office tomorrow but I’ll probably still check in some time. I can always remote in and send you something.

    Thanks!

    Cheers,

    Joe



  • @Tom-Elliott

    Will do and I’ll update. It happens even without snapins running. But I’ll update and see if it continues.

    Thanks!


  • Senior Developer

    @Joe-Gill please try updating too.



  • @Developers

    Well, I’m back at it again. I noticed that this issue also occurs on unicasts. I deployed several machines yesterday; pushing each machine as I added them to FOG. I had about half of those machines where the task never completed in the “Active Task” screen and went into reboot hell.

    I had a snapin addon on these guys. I’m going to try re-pushing them without said plugin and see what happens.

    Cheers,

    Joe



  • @Tom-Elliott
    Ok. So the fix I have is to cancel the active task for the problem host in task manager. Than, I go into the MYSQL database on the server and clear out the task there. I then stop and restart the multi tasking service. Then things seem to be fine for the next session.

    @Wayne-Workman Here is the Apache2 Error.log file for today. I did notice their was a fatal error the day before. It appears that this same entry happened many times that day.

    [Tue Jul 26 20:49:35.416710 2016] [:error] [pid 4097] [client 172.16.19.149:65194] PHP Fatal error:  Uncaught exception 'Exception' with message 'Window size must be g$
    [Tue Jul 26 20:49:36.423986 2016] [:error] [pid 4100] [client 172.16.19.149:65197] PHP Fatal error:  Uncaught exception 'Exception' with message 'Window size must be g$
    [Wed Jul 27 07:26:32.280481 2016] [:error] [pid 4201] [client 172.16.16.150:59389] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 07:26:48.354429 2016] [:error] [pid 4201] [client 172.16.16.150:59404] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 07:27:01.053034 2016] [:error] [pid 4201] [client 172.16.16.150:59417] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 07:36:57.647527 2016] [:error] [pid 7834] [client 172.16.16.150:60058] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 07:37:15.662213 2016] [:error] [pid 4206] [client 172.16.16.150:60082] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 07:37:28.563481 2016] [:error] [pid 4097] [client 172.16.16.150:60101] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 08:30:45.361208 2016] [:error] [pid 9327] [client 172.16.16.150:62604] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 09:10:51.246977 2016] [:error] [pid 4201] [client 172.16.16.150:64132] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 09:11:10.725368 2016] [:error] [pid 6575] [client 172.16.16.150:64148] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 09:11:19.335300 2016] [:error] [pid 7905] [client 172.16.16.150:64159] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 09:12:35.911531 2016] [:error] [pid 4101] [client 172.16.16.150:64236] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 11:16:54.796082 2016] [:error] [pid 4206] [client 172.16.16.150:52767] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 11:17:07.981127 2016] [:error] [pid 4201] [client 172.16.16.150:52780] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 12:58:17.660609 2016] [:error] [pid 11833] [client 172.16.18.89:49845] PHP Strict Standards:  Only variables should be passed by reference in /var/www/html$
    [Wed Jul 27 15:01:59.879251 2016] [:error] [pid 13392] [client 172.16.16.150:62040] PHP Strict Standards:  Only variables should be passed by reference in /var/www/htm$
    [Wed Jul 27 15:02:17.611327 2016] [:error] [pid 13023] [client 172.16.16.150:62056] PHP Strict Standards:  Only variables should be passed by reference in /var/www/htm$
    [Wed Jul 27 15:53:39.483338 2016] [:error] [pid 4201] [client 172.16.16.150:63406] PHP Warning:  pack(): Type H: illegal hex digit K in /var/www/html/fog/lib/fog/fogba$
    

  • Senior Developer

    @Tom-Elliott and by that I mean will be fixed.


  • Senior Developer

    I’ve confirmed the non closing multicast task. I still don’t understand what you mean by “I kill the taskings” Which tasking are you referring to? The host’s individual tasking, or the multicast tasking?

    THe non-closing multicast will be available for RC-5.


  • Moderator

    @Joe-Gill apache error logs from the time at the end of imaging to when the reboots start happening would be very informative.



  • @Tom-Elliott 1 in 5 get stuck in reboot loop AND they also are the ones that fail to complete taskings. I kill the taskings and reboot loop ceases and operations complete as normal.


  • Senior Developer

    @Joe-Gill What I mean, 1 in 5 hosts fail to complete their taskings? Or 1 in 5 get stuck in reboot loop?

    If the tasking is still present under the “active tasks” page (not active multicast) the reboot would cause those systems to boot into the Imaging side of things.



  • @Tom-Elliott Exactly. Only 1 in 5 it seems. But it’s identical hardware and images all around.


  • Senior Developer

    @Joe-Gill So it’s not ALL hosts in the multicast task?



  • @Tom-Elliott

    Yes the hosts that go into reboot hell still exist in Task Management Page. All of the other hosts in that session seem to finish just fine.


  • Senior Developer

    I don’t understand. If you go to the Task Management Page, (not multicast tasks) do the tasks still exist?




Log in to reply
 

390
Online

38748
Users

10575
Topics

100102
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.