Fog Client service stops responding

Joe Schmitt

@tonytonyistony said in Fog Client service stops responding:

can no longer successfully run the service without a reboot. The only way to restore connectivity to the FOG server from the client host is to reboot the OS and have the service start at boot.

First off thank you for gathering all the information ahead of time, it definitely helps us out.
Does the client stop entirely, or are the logs essentially all:

 3/31/2017 10:12 AM Middleware::Response ERROR: Unable to get subsection
 3/31/2017 10:12 AM Middleware::Response ERROR: Object reference not set to an instance of an object

tonytonyistony

Joe

Thank you for the accolades. Im in tech support myself, and I know the feeling of spending more time collecting data, than actually troubleshooting.

I just wanted to respond that I got your message, and once I get in to the office on Monday I can have a much more detailed dialog. I could even grant remote access to the device.

But to briefly answer your question…

"Does the client stop entirely,?"

Yeah it basically becomes unresponsive to the Server. We issue a reboot/re-image from the server to the client and it does nothing. Once I go back and check the logs on the client all I see is what I attached above. Once I reboot and check the logs, they look perfect and the server can communicate.

With my experience using Linux, this part concerns me. However, it doesn’t explain that it works after boot.

3/31/2017 10:12 AM Middleware::Communication URL: http://10.10.100.252/fog/management/index.php?sub=requestClientInfo&mac=C4:6E:1F:04:54:27|B8:AC:6F:85:32:6D|52:54:00:71:D1:FE|52:54:00:71:D1:FE|B8:AC:6F:85:32:6D|52:54:00:9F:A0:77|52:54:00:2D:A7:62|52:54:00:2D:A7:62|FE:00:DD:C0:A1:06|FE:54:00:D0:9F:64|FE:00:DD:C0:A1:65|FE:54:00:7C:BC:97|FE:06:0A:0E:A1:00|FE:06:0A:0E:A1:01|FE:06:0A:0E:A1:02|FE:06:0A:0E:A1:03|52:54:00:51:61:E8|52:54:00:51:61:E8|FE:00:DD:C0:B1:07|FE:54:00:86:86:3E|FE:00:DD:C0:B1:66|FE:54:00:FB:99:70|FE:06:0A:0E:B1:00|FE:06:0A:0E:B1:01|FE:06:0A:0E:B1:02|FE:06:0A:0E:B1:03|FE:06:0A:0E:B1:04|FE:06:0A:0E:B1:05|52:54:00:A7:08:1F|52:54:00:A7:08:1F|FE:00:DD:C0:C1:08|FE:54:00:98:34:A7|FE:00:DD:C0:C1:67|FE:54:00:1E:E7:85|FE:06:0A:0E:C1:00|FE:06:0A:0E:C1:01|FE:06:0A:0E:C1:02|FE:06:0A:0E:C1:03|FE:06:0A:0E:C1:04|FE:06:0A:0E:C1:05|52:54:00:C2:F6:BD|52:54:00:C2:F6:BD|FE:00:DD:C0:D1:09|FE:54:00:A2:37:BE|FE:00:DD:C0:D1:68|FE:54:00:0A:C1:A7|FE:06:0A:0E:D1:00|FE:06:0A:0E:D1:01|FE:06:0A:0E:D1:02|FE:06:0A:0E:D1:03|FE:54:00:A8:95:EB|FE:54:00:53:C0:10|FE:54:00:AD:1C:98|FE:54:00:1A:59:4F|FE:54:00:43:73:5C|FE:54:00:F3:69:66|FE:06:0A:0E:A1:02|FE:06:0A:0E:A1:03|FE:06:0A:0E:D1:03|FE:06:0A:0E:D1:02|FE:06:0A:0E:A1:00|FE:06:0A:0E:A1:01|FE:06:0A:0E:B1:04|FE:06:0A:0E:B1:05|FE:06:0A:0E:C1:00|FE:06:0A:0E:C1:01|FE:06:0A:0E:B1:02|FE:06:0A:0E:C1:02&newService&json
3/31/2017 10:12 AM Middleware::Response ERROR: Could not parse data
3/31/2017 10:12 AM Middleware::Response ERROR: Unexpected character encountered while parsing value: T. Path '', line 0, position 0.

I wonder if some code was deprecated when Ubuntu 16.04.1 updated to .2

Joe Schmitt

@tonytonyistony Yeah it basically becomes unresponsive to the Server., does the client continue to log, or does the process itself just stop. I ask because if it crashes completely (the process dies), then its likely an OS-specific issue. Whereas if the client continues to log but is unresponsive to the server, then its possible that an irrecoverable bad state does exist in the client’s code, and that needs to be patched.

As for these messages:

Mar 31 10:09:51 Pascua-02 systemd[1]: FOGService.service: Unit entered failed state.
Mar 31 10:09:51 Pascua-02 systemd[1]: FOGService.service: Failed with result 'timeout'.

they are likely due to how we wrap the client on Linux. Keep in mind we are actually making a Windows oriented service run under every common *nix based daemon / service system. It’s likely that Ubuntu’s daemon system does not like how our service wrappers handle things, but it definitely won’t harm anything or cause any failed code. We could probably fix the client’s service wrapper for Ubuntu, but we’re a tad stretched for developers right now and its a fairly low priority ticket.

tonytonyistony

@Joe-Schmitt

does the client continue to log, or does the process itself just stop.

It continues to log, repeating what I attached into the forum post until I reboot.

Joe Schmitt

@tonytonyistony while the client is in this bad state, (continually logging Could not parse data), could you try going to the host in the web portal and clicking Reset Encryption Data? Its not needed, but I want to see if it’ll force the client to re-authenticate and fix this issue itself.

tonytonyistony

@Joe-Schmitt said in Fog Client service stops responding:

@tonytonyistony while the client is in this bad state, (continually logging Could not parse data), could you try going to the host in the web portal and clicking Reset Encryption Data? Its not needed, but I want to see if it’ll force the client to re-authenticate and fix this issue itself.

This is the log output after hitting the “Reset Encryption Data” button in the server.
I did not do anything else but cat the “fog.log” file. Is there anything else you wanted me to do on the client side after hitting the reset button on the server?

#Fog.log
------------------------------------------------------------------------------
----------------------------------UserTracker---------------------------------
------------------------------------------------------------------------------
 4/5/2017 9:58 AM Client-Info Client Version: 0.11.8
 4/5/2017 9:58 AM Client-Info Client OS:      Linux
 4/5/2017 9:58 AM Client-Info Server Version: 689680
 4/5/2017 9:58 AM Middleware::Response ERROR: Unable to get subsection
 4/5/2017 9:58 AM Middleware::Response ERROR: Object reference not set to an instance of an object
 4/5/2017 9:58 AM Service Sleeping for 125 seconds
 4/5/2017 10:01 AM Middleware::Communication URL: http://10.10.100.252/fog/management/index.php?sub=requestClientInfo&configure&newService&json
 4/5/2017 10:01 AM Middleware::Response Success
 4/5/2017 10:01 AM Middleware::Communication Download: http://10.10.100.252/fog/management/other/fogimg.png
 4/5/2017 10:01 AM Middleware::Communication URL: http://10.10.100.252/fog/management/index.php?sub=requestClientInfo&mac=C4:6E:1F:04:54:27|B8:AC:6F:85:32:6D|52:54:00:71:D1:FE|52:54:00:71:D1:FE|B8:AC:6F:85:32:6D|52:54:00:9F:A0:77|52:54:00:2D:A7:62|52:54:00:2D:A7:62|FE:00:DD:C0:A1:06|FE:54:00:D0:9F:64|FE:00:DD:C0:A1:65|FE:54:00:7C:BC:97|FE:06:0A:0E:A1:00|FE:06:0A:0E:A1:01|FE:06:0A:0E:A1:02|FE:06:0A:0E:A1:03|52:54:00:51:61:E8|52:54:00:51:61:E8|FE:00:DD:C0:B1:07|FE:54:00:86:86:3E|FE:00:DD:C0:B1:66|FE:54:00:FB:99:70|FE:06:0A:0E:B1:00|FE:06:0A:0E:B1:01|FE:06:0A:0E:B1:02|FE:06:0A:0E:B1:03|FE:06:0A:0E:B1:04|FE:06:0A:0E:B1:05|52:54:00:A7:08:1F|52:54:00:A7:08:1F|FE:00:DD:C0:C1:08|FE:54:00:98:34:A7|FE:00:DD:C0:C1:67|FE:54:00:1E:E7:85|FE:06:0A:0E:C1:00|FE:06:0A:0E:C1:01|FE:06:0A:0E:C1:02|FE:06:0A:0E:C1:03|FE:06:0A:0E:C1:04|FE:06:0A:0E:C1:05|52:54:00:C2:F6:BD|52:54:00:C2:F6:BD|FE:00:DD:C0:D1:09|FE:54:00:A2:37:BE|FE:00:DD:C0:D1:68|FE:54:00:0A:C1:A7|FE:06:0A:0E:D1:00|FE:06:0A:0E:D1:01|FE:06:0A:0E:D1:02|FE:06:0A:0E:D1:03|FE:54:00:A8:95:EB|FE:54:00:53:C0:10|FE:54:00:AD:1C:98|FE:54:00:1A:59:4F|FE:54:00:43:73:5C|FE:54:00:F3:69:66|FE:06:0A:0E:A1:02|FE:06:0A:0E:A1:03|FE:06:0A:0E:D1:03|FE:06:0A:0E:D1:02|FE:06:0A:0E:A1:00|FE:06:0A:0E:A1:01|FE:06:0A:0E:B1:04|FE:06:0A:0E:B1:05|FE:06:0A:0E:C1:00|FE:06:0A:0E:C1:01|FE:06:0A:0E:B1:02|FE:06:0A:0E:C1:02&newService&json
 4/5/2017 10:01 AM Middleware::Response ERROR: Could not parse data
 4/5/2017 10:01 AM Middleware::Response ERROR: Unexpected character encountered while parsing value: T. Path '', line 0, position 0.
 4/5/2017 10:01 AM Middleware::Communication URL: http://10.10.100.252/fog/service/getversion.php?clientver&newService&json
 4/5/2017 10:01 AM Middleware::Communication URL: http://10.10.100.252/fog/service/getversion.php?newService&json
 4/5/2017 10:01 AM Service ERROR: Unable to get cycle data
 4/5/2017 10:01 AM Service ERROR: Object reference not set to an instance of an object
 4/5/2017 10:01 AM Middleware::Response Success

------------------------------------------------------------------------------
---------------------------------ClientUpdater--------------------------------
------------------------------------------------------------------------------
 4/5/2017 10:01 AM Client-Info Client Version: 0.11.8
 4/5/2017 10:01 AM Client-Info Client OS:      Linux
 4/5/2017 10:01 AM Client-Info Server Version: 689680
 4/5/2017 10:01 AM Middleware::Response ERROR: Unable to get subsection
 4/5/2017 10:01 AM Middleware::Response ERROR: Object reference not set to an instance of an object

------------------------------------------------------------------------------
----------------------------------TaskReboot----------------------------------
------------------------------------------------------------------------------
 4/5/2017 10:01 AM Client-Info Client Version: 0.11.8
 4/5/2017 10:01 AM Client-Info Client OS:      Linux
 4/5/2017 10:01 AM Client-Info Server Version: 689680
 4/5/2017 10:01 AM Middleware::Response ERROR: Unable to get subsection
 4/5/2017 10:01 AM Middleware::Response ERROR: Object reference not set to an instance of an object

------------------------------------------------------------------------------
--------------------------------HostnameChanger-------------------------------
------------------------------------------------------------------------------
 4/5/2017 10:01 AM Client-Info Client Version: 0.11.8
 4/5/2017 10:01 AM Client-Info Client OS:      Linux
 4/5/2017 10:01 AM Client-Info Server Version: 689680
 4/5/2017 10:01 AM Middleware::Response ERROR: Unable to get subsection
 4/5/2017 10:01 AM Middleware::Response ERROR: Object reference not set to an instance of an object

------------------------------------------------------------------------------
---------------------------------SnapinClient---------------------------------
------------------------------------------------------------------------------
 4/5/2017 10:01 AM Client-Info Client Version: 0.11.8
 4/5/2017 10:01 AM Client-Info Client OS:      Linux
 4/5/2017 10:01 AM Client-Info Server Version: 689680
 4/5/2017 10:01 AM Middleware::Response ERROR: Unable to get subsection
 4/5/2017 10:01 AM Middleware::Response ERROR: Object reference not set to an instance of an object

------------------------------------------------------------------------------
--------------------------------PrinterManager--------------------------------
------------------------------------------------------------------------------
 4/5/2017 10:01 AM Client-Info Client Version: 0.11.8
 4/5/2017 10:01 AM Client-Info Client OS:      Linux
 4/5/2017 10:01 AM Client-Info Server Version: 689680
 4/5/2017 10:01 AM Middleware::Response ERROR: Unable to get subsection
 4/5/2017 10:01 AM Middleware::Response ERROR: Object reference not set to an instance of an object

------------------------------------------------------------------------------
--------------------------------PowerManagement-------------------------------
------------------------------------------------------------------------------
 4/5/2017 10:01 AM Client-Info Client Version: 0.11.8
 4/5/2017 10:01 AM Client-Info Client OS:      Linux
 4/5/2017 10:01 AM Client-Info Server Version: 689680
 4/5/2017 10:01 AM Middleware::Response ERROR: Unable to get subsection
 4/5/2017 10:01 AM Middleware::Response ERROR: Object reference not set to an instance of an object

------------------------------------------------------------------------------
----------------------------------UserTracker---------------------------------
------------------------------------------------------------------------------
 4/5/2017 10:01 AM Client-Info Client Version: 0.11.8
 4/5/2017 10:01 AM Client-Info Client OS:      Linux
 4/5/2017 10:01 AM Client-Info Server Version: 689680
 4/5/2017 10:01 AM Middleware::Response ERROR: Unable to get subsection
 4/5/2017 10:01 AM Middleware::Response ERROR: Object reference not set to an instance of an object
 4/5/2017 10:01 AM Service Sleeping for 87 seconds

#syslog
#No new entries

Joe Schmitt

@tonytonyistony Nothing else is needed after hitting reset encryption button, the client should have automatically recovered from the failed state. I’ll send you a developer build of the client later today, if you could install it on a single problematic host that would be extremely helpful as it should log what’s causing the error. However, keep in mind developer builds of the client are extremely verbose and not meant for production use as it will likely log sensitive information.

tonytonyistony

@Joe-Schmitt said in Fog Client service stops responding:

but I want to see if it’ll force the client to re-authenticate and fix this issue itself.

I wanted to say this is the easiest stop gap in my opinion. I could make a cron script that detects the error, then issue some sort of restart, then I wouldn’t need user intervention to fix the issue.

tonytonyistony

@Joe-Schmitt said in Fog Client service stops responding:

@tonytonyistony Nothing else is needed after hitting reset encryption button, the client should have automatically recovered from the failed state. I’ll send you a developer build of the client later today, if you could install it on a single problematic host that would be extremely helpful as it should log what’s causing the error. However, keep in mind developer builds of the client are extremely verbose and not meant for production use as it will likely log sensitive information.

Awesome! Just let me know what needs to be done and Ill make it happen and get back to you.

tonytonyistony

@Joe-Schmitt said in Fog Client service stops responding:

@tonytonyistony Nothing else is needed after hitting reset encryption button, the client should have automatically recovered from the failed state. I’ll send you a developer build of the client later today, if you could install it on a single problematic host that would be extremely helpful as it should log what’s causing the error. However, keep in mind developer builds of the client are extremely verbose and not meant for production use as it will likely log sensitive information.

Hey Joe. So Im back at trying to figure this FOG client out in Linux. I never did receive any information from you regarding this dev build. Has there been any update or progress made getting this client working in Linux since our last discussion?

Im very confident this is due to PhP and deprecated code. Is it possible you could give me the PhP version you guys programmed in so I can try installing the right PhP version and see if that makes any positive effect?

tonytonyistony

@tonytonyistony said in Fog Client service stops responding:

3/31/2017 10:12 AM Middleware::Communication URL: http://10.10.100.252/fog/management/index.php?sub=requestClientInfo&mac=C4:6E:1F:04:54:27|B8:AC:6F:85:32:6D|52:54:00:71:D1:FE|52:54:00:71:D1:FE|B8:AC:6F:85:32:6D|52:54:00:9F:A0:77|52:54:00:2D:A7:62|52:54:00:2D:A7:62|FE:00:DD:C0:A1:06|FE:54:00:D0:9F:64|FE:00:DD:C0:A1:65|FE:54:00:7C:BC:97|FE:06:0A:0E:A1:00|FE:06:0A:0E:A1:01|FE:06:0A:0E:A1:02|FE:06:0A:0E:A1:03|52:54:00:51:61:E8|52:54:00:51:61:E8|FE:00:DD:C0:B1:07|FE:54:00:86:86:3E|FE:00:DD:C0:B1:66|FE:54:00:FB:99:70|FE:06:0A:0E:B1:00|FE:06:0A:0E:B1:01|FE:06:0A:0E:B1:02|FE:06:0A:0E:B1:03|FE:06:0A:0E:B1:04|FE:06:0A:0E:B1:05|52:54:00:A7:08:1F|52:54:00:A7:08:1F|FE:00:DD:C0:C1:08|FE:54:00:98:34:A7|FE:00:DD:C0:C1:67|FE:54:00:1E:E7:85|FE:06:0A:0E:C1:00|FE:06:0A:0E:C1:01|FE:06:0A:0E:C1:02|FE:06:0A:0E:C1:03|FE:06:0A:0E:C1:04|FE:06:0A:0E:C1:05|52:54:00:C2:F6:BD|52:54:00:C2:F6:BD|FE:00:DD:C0:D1:09|FE:54:00:A2:37:BE|FE:00:DD:C0:D1:68|FE:54:00:0A:C1:A7|FE:06:0A:0E:D1:00|FE:06:0A:0E:D1:01|FE:06:0A:0E:D1:02|FE:06:0A:0E:D1:03|FE:54:00:A8:95:EB|FE:54:00:53:C0:10|FE:54:00:AD:1C:98|FE:54:00:1A:59:4F|FE:54:00:43:73:5C|FE:54:00:F3:69:66|FE:06:0A:0E:A1:02|FE:06:0A:0E:A1:03|FE:06:0A:0E:D1:03|FE:06:0A:0E:D1:02|FE:06:0A:0E:A1:00|FE:06:0A:0E:A1:01|FE:06:0A:0E:B1:04|FE:06:0A:0E:B1:05|FE:06:0A:0E:C1:00|FE:06:0A:0E:C1:01|FE:06:0A:0E:B1:02|FE:06:0A:0E:C1:02&newService&json

This is the problem. The machine Im trying to image has many virtual machines running on it, which all those MAC addresses are all the interfaces being used by QEMU and VIRSH. How can I get the fog client to not send all those MAC addresses?

EDIT: When we entered the information below in a browser we got a response from JSON

Working
http://10.10.100.252/fog/management/index.php?sub=requestClientInfo&mac=C4:6E:1F:04:54:27&newService&json
JSON Response: #!ihc

Not working
JSON Response: Too many MACs

Sebastian Roth

@tonytonyistony said:

When we entered the information below in a browser we got a response from JSON

Did you try the setting “Ignore MAC on client” (checkbox next to the MAC addresses) in the host settings on the FOG web GUI?

tonytonyistony

@Sebastian-Roth
Yes. I just did this and the client still attempts to send every single MAC address to the server.

Sebastian Roth

@tonytonyistony I am really sorry this took so long although it turns out to be really simple for you to fix. Getting to understand the client code and actually replicating the issue took me hell of a lot of time compared to the “fix”.

The Too many MACs was a good catch of yours although I couldn’t reproduce for hours as newer versions of FOG (web UI not the client) don’t have that issue anymore and I kept looking at the wrong places. But to make a long story short, the issue for you as I see it is that the Too many MACs is not returned as JSON but as normal string which the client does not like.

So there are two simple options for you:

Set FOG_QUICKREG_MAX_PENDING_MACS in the FOG Settings on the web UI to a value high enough for all the MACs sent by the client. I guess 100 would do you for now. Note that even if you approve(d) all the MACs for this client in the web UI the number needs to be high enough. I think this is true even in most current code and needs fixing @Tom-Elliott?
The real fix in the code is really simple as well. Just edit /var/www/fog/lib/client/registerclient.class.php and find this part of the code (line numbers in the front so you hopefully find it easily:

 99        if (count($MACs) > $maxPending + 1) {
100            throw new Exception(_('Too many MACs'));
101        }

Be aware this is in there two times, one for the new client (further up) and one for the old client (more to the bottom). You want to change only the top one! Change it and save the file:

 99         if (count($MACs) > $maxPending + 1) {
100            return array('error' => _('Too many MACs'));
101         }

No need to restart anything. Even your FOG client should recover on its own on the next request!!

Sebastian Roth

@tonytonyistony Any news on this one?

Fog Client service stops responding

307

11.7k

17.1k

154.7k