Windows 10 - FOG Client Failing To Restart

dclark

Server
FOG Version: 1.4.4
OS: Ubuntu 16.04

Client
Service Version: 0.11.12
OS: Windows 10 Education 64bit

I have had a Windows 10 (1709 Education) deploying since November without issue, auto domain join, snapin install, etc. (Approximately 1200 deployments)

Suddenly I now have a situation where after the image completes and Windows sets up the expected domain joining occurs and restarts the machine. Issue is that after it reboots the FOG Service never restarts even though it is set to automatic. Multiple reboots end with the same situation. If you manually start the FOG Service again it works fine and will be started on the next reboot.

I am thinking it is something in Windows 10 wanting to update but am curious if anyone else has/is experiencing this and if you have found a solution?

Log from the initial startup:

 1/8/2018 2:02 PM Main Overriding exception handling
 1/8/2018 2:02 PM Main Bootstrapping Zazzles
 1/8/2018 2:02 PM Controller Initialize
 1/8/2018 2:02 PM Controller Start

 1/8/2018 2:02 PM Service Starting service
 1/8/2018 2:02 PM Bus Became bus server
 1/8/2018 2:02 PM Bus Emmiting message on channel: Status
 1/8/2018 2:02 PM Service Invoking early JIT compilation on needed binaries

------------------------------------------------------------------------------
--------------------------------Authentication--------------------------------
------------------------------------------------------------------------------
 1/8/2018 2:02 PM Client-Info Version: 0.11.12
 1/8/2018 2:02 PM Client-Info OS:      Windows
 1/8/2018 2:02 PM Middleware::Authentication Waiting for authentication timeout to pass
 1/8/2018 2:02 PM Middleware::Communication Download: http://10.40.25.165/fog/management/other/ssl/srvpublic.crt
 1/8/2018 2:02 PM Data::RSA FOG Server CA cert found
 1/8/2018 2:02 PM Middleware::Authentication Cert OK
 1/8/2018 2:02 PM Middleware::Authentication ERROR: Could not get security token
 1/8/2018 2:02 PM Middleware::Authentication ERROR: Could not find file 'C:\Program Files (x86)\FOG\token.dat'.
 1/8/2018 2:02 PM Middleware::Communication POST URL: http://10.40.25.165/fog/management/index.php?sub=requestClientInfo&authorize&newService
 1/8/2018 2:02 PM Middleware::Response Success
 1/8/2018 2:02 PM Middleware::Authentication Authenticated


 1/8/2018 2:02 PM Middleware::Communication URL: http://10.40.25.165/fog/management/index.php?sub=requestClientInfo&configure&newService&json
 1/8/2018 2:02 PM Middleware::Response Success
 1/8/2018 2:02 PM Middleware::Communication Download: http://10.40.25.165/fog/management/other/SMALL.png
 1/8/2018 2:02 PM Middleware::Communication URL: http://10.40.25.165/fog/management/index.php?sub=requestClientInfo&mac=48:4D:7E:DA:32:67&newService&json
 1/8/2018 2:02 PM Middleware::Response Success
 1/8/2018 2:02 PM Middleware::Communication URL: http://10.40.25.165/fog/service/getversion.php?clientver&newService&json
 1/8/2018 2:02 PM Middleware::Communication URL: http://10.40.25.165/fog/service/getversion.php?newService&json

 1/8/2018 2:02 PM Service Creating user agent cache
 1/8/2018 2:02 PM Middleware::Response Invalid time
 1/8/2018 2:02 PM Middleware::Response Module is disabled globally on the FOG server
 1/8/2018 2:02 PM Middleware::Response Module is disabled globally on the FOG server
 1/8/2018 2:02 PM Service Initializing modules

------------------------------------------------------------------------------
---------------------------------ClientUpdater--------------------------------
------------------------------------------------------------------------------
 1/8/2018 2:02 PM Client-Info Client Version: 0.11.12
 1/8/2018 2:02 PM Client-Info Client OS:      Windows
 1/8/2018 2:02 PM Client-Info Server Version: 1.4.4
 1/8/2018 2:02 PM Middleware::Response Success
------------------------------------------------------------------------------


------------------------------------------------------------------------------
----------------------------------TaskReboot----------------------------------
------------------------------------------------------------------------------
 1/8/2018 2:02 PM Client-Info Client Version: 0.11.12
 1/8/2018 2:02 PM Client-Info Client OS:      Windows
 1/8/2018 2:02 PM Client-Info Server Version: 1.4.4
 1/8/2018 2:02 PM Middleware::Response Success
------------------------------------------------------------------------------


------------------------------------------------------------------------------
--------------------------------HostnameChanger-------------------------------
------------------------------------------------------------------------------
 1/8/2018 2:02 PM Client-Info Client Version: 0.11.12
 1/8/2018 2:02 PM Client-Info Client OS:      Windows
 1/8/2018 2:02 PM Client-Info Server Version: 1.4.4
 1/8/2018 2:02 PM Middleware::Response Success
 1/8/2018 2:02 PM HostnameChanger Checking Hostname
 1/8/2018 2:02 PM HostnameChanger Renaming host to SDSA-LAB104-22
 1/8/2018 2:02 PM HostnameChanger Joining domain
 1/8/2018 2:02 PM HostnameChanger Success, code =  0
 1/8/2018 2:02 PM Power Creating shutdown request
 1/8/2018 2:02 PM Power Parameters: /r /c "Indian River School District needs to rename your computer" /t 0
 1/8/2018 2:02 PM Bus Emmiting message on channel: Power
 1/8/2018 2:02 PM Power Attempt 1/6 to shutdown computer
 1/8/2018 2:02 PM Power --> API call returned 1, will re-attempt in 5 minutes

Joe Schmitt

@dclark v0.11.13 of the client (in the newest RC) patches the win1709 issues, thank you for your help debugging this!

Joe Schmitt

Linking with https://forums.fogproject.org/topic/11305/fogclient-0-11-12-windows-10-1709-reboot-fails-after-joining-domain as @TrialAndError has the same issue. It seems to be specific to 1709.

dclark

Thank You for the link - Yes, it seems very similar. Also looking at the timing of the post, about a week ago, I believe that my issue began appearing at almost the same time. I am thinking that the latest RollUp from Microsoft is the culprit and possibly running in the background when the Fog Client is trying to perform its duties but have not been able to prove it. The machines I am having this issue are restarting and joining the domain but the Fog Client is not starting again after that.

It is possible that the actual restart is being performed by Windows update or another process. Does the Fog Client maintain some type of flag that it is waiting for a restart that may not be getting reset when the restart comes from another process and then hanging the Fog process on the next startup? The downside of this theory is that I have created a new image and installed all available updates prior to capture and am getting the same behavior with it.

dclark

As a follow up, I am starting to see the Fog Client issues on machines that have been imaged for quite some time. It is starting to look like an error within the latest patches from MS. This is the tail of the log that is starting to show up on existing machines with stopped clients.

------------------------------------------------------------------------------
----------------------------------UserTracker---------------------------------
------------------------------------------------------------------------------
 1/10/2018 8:58 PM Client-Info Client Version: 0.11.12
 1/10/2018 8:58 PM Client-Info Client OS:      Windows
 1/10/2018 8:58 PM Client-Info Server Version: 1.4.4
 1/10/2018 8:58 PM Middleware::Response Success
 1/10/2018 8:59 PM User ERROR: Unable to get logged in users
 1/10/2018 8:59 PM User ERROR: Call was canceled by the message filter. (Exception from HRESULT: 0x80010002 (RPC_E_CALL_CANCELED))
------------------------------------------------------------------------------

 1/10/2018 8:59 PM Service Sleeping for 303 seconds
 1/10/2018 9:01 PM User ERROR: Unable to get logged in users
 1/10/2018 9:01 PM User ERROR: Call was canceled by the message filter. (Exception from HRESULT: 0x80010002 (RPC_E_CALL_CANCELED))
 1/10/2018 9:01 PM Power Creating shutdown request
 1/10/2018 9:01 PM Power Parameters: /s /c "FOG PowerManagement" /t 0
 1/10/2018 9:01 PM Bus Emmiting message on channel: Power
 1/10/2018 9:01 PM Log Unhandled exception caught
 1/10/2018 9:01 PM Log     Terminating: True
 1/10/2018 9:01 PM Log     Hash code: 34170346

Also catching an error related to .net on a few if these machines when trying to re-install the client.

alt text

Joe Schmitt

@dclark can you try reinstalling the .NET framework? I agree this is starting to seem more and more like a bad Windows update, hopefully we can isolate which update is causing this.

Joe Schmitt

@dclark some thoughts:

Could you right-click -> Properties on C:\Windows\Microsoft.Net\Framework\v4.0.30319\mscoreei.dll, and check out the Created, Modified, and also the Digital Signatures tab to make sure the file wasn’t overwritten?
Can you please run sfc /scannow from an administrative CMD? (this is the Windows System File Checker)
Run a full virus scan. It’s likely not a virus, but I just want to make sure we eliminate that possibility

dclark

@joe-schmitt Everything as far as the .NET framework looks legit and file scan finds nothing of consequence.

I really believe at this point the issue is caused by the Jan 3 update by Microsoft and the Spectre patch. There is a newly released update to disable parts of it which I have applied to problematic machines and also installed in a test image with initial success.

Guidance - https://support.microsoft.com/en-us/help/4078130/update-to-disable-mitigation-against-spectre-variant-2

KB4078130 - https://www.catalog.update.microsoft.com/Search.aspx?q=KB4078130

dclark

@joe-schmitt Looks like this has not been totally eradicated. It shows up randomly on existing machines and on freshly imaged machines. What registry keys are set/reset when the client is installed/started? It is possible that they are getting wiped out or reset due to a Windows update/patch issue.

Joe Schmitt

@dclark v0.11.13 of the client (in the newest RC) patches the win1709 issues, thank you for your help debugging this!

Windows 10 - FOG Client Failing To Restart

253

11.6k

17.1k

154.6k