FOG client on CentOS can't authenticate/not working

[mechalas]

Server
Ubuntu 16.04.5 LTS
4.4.98 kernel
Fog Server v1.5.5

Client
CentOS 7.5
3.10.0-957 kernel
Fog Client 0.11.16

On client, /opt/fog-service/fog.log at startup:

 2/27/2019 12:26 PM Main Overriding exception handling
 2/27/2019 12:26 PM Main Bootstrapping Zazzles
 2/27/2019 12:26 PM Controller Initialize
 2/27/2019 12:26 PM Controller Start

 2/27/2019 12:26 PM Service Starting service
 2/27/2019 12:26 PM Bus Became bus server
 2/27/2019 12:26 PM Bus Emmiting message on channel: Status
 2/27/2019 12:26 PM Service Invoking early JIT compilation on needed binaries

------------------------------------------------------------------------------
--------------------------------Authentication--------------------------------
------------------------------------------------------------------------------
 2/27/2019 12:26 PM Client-Info Version: 0.11.16
 2/27/2019 12:26 PM Client-Info OS:      Linux
 2/27/2019 12:26 PM Middleware::Authentication Waiting for authentication timeout to pass
 2/27/2019 12:26 PM Middleware::Communication Download: http://XXXXX/fog/management/other/ssl/srvpublic.crt
 2/27/2019 12:26 PM Middleware::Authentication ERROR: Could not authenticate
 2/27/2019 12:26 PM Middleware::Authentication ERROR: Value cannot be null.
Parameter name: authority

Where XXXXX is a FQDN.

My Ubuntu 16.04.5 and 18.04.2 clients are working, so I suspect this may be an issue with mono packaging for CentOS. I used the instructions here:

https://wiki.fogproject.org/wiki/index.php?title=FOG_Client#CentOS_7.2C_Fedora_19.2B.2C_and_derivatives

There are no errors in the apache error log.

Access log shows HTTP OK response codes:

X.X.X.X:80 n.n.n.n - - [27/Feb/2019:12:26:35 -0800] "GET /fog/management/other/ca.cert.der HTTP/1.1" 200 1525 "-" "-"
X.X.X.X:80 n.n.n.n - - [27/Feb/2019:12:26:37 -0800] "GET /fog/management/other/ssl/srvpublic.crt HTTP/1.1" 200 1958 "-" "-"
X.X.X.X:80 n.n.n.n - - [27/Feb/2019:12:36:37 -0800] "GET /fog/management/other/ssl/srvpublic.crt HTTP/1.1" 200 1958 "-" "-"
X.X.X.X:80 n.n.n.n - - [27/Feb/2019:12:36:37 -0800] "GET /fog/service/getversion.php?clientver&newService&json HTTP/1.1" 200 554 "-" "-"
X.X.X.X:80 n.n.n.n - - [27/Feb/2019:12:36:37 -0800] "GET /fog/service/getversion.php?newService&json HTTP/1.1" 200 552 "-" "-"
X.X.X.X:80 n.n.n.n - - [27/Feb/2019:12:34:37 -0800] "GET /fog/management/index.php?sub=requestClientInfo&mac=OMITTED&newService&json HTTP/1.1" 200 552 "-" "-"

Where X.X.X.X is the server IP and n.n.n.n is the client IP. tcpdump shows that the public cert is getting transferred in its entirety. The MAC address for requestClientInfo is correct.

[mechalas]

I should add that this is a fresh install of CentOS 7.5.

[mechalas]

I see the same problem on Fedora 27, which uses the same mono repo as CentOS 7.

[Sebastian Roth]

@mechalas You’ve probably searched the forums and found similar posts like this: https://forums.fogproject.org/topic/10788/authentication-error-could-not-authenticate-value-cannot-be-null

We see this happing with broken Mono certificate stores. Try uninstalling the client, removing Mono and installing an older Mono build, and then installing the client.

[mechalas]

@Sebastian-Roth Nope, I was searching on other terms and hadn’t seen this one yet.

I’ll give it a shot.

[Sebastian Roth]

@mechalas Keeping my fingers crossed you can solve it this way. Our fog-client developer has talked about hoping to fix this issue by using different means of storing the certificates on the client but he’s way to busy to work on this project and I am fairly sure a new fog-client version with such a huge change won’t come any time soon. Just so you know.

[mechalas]

Doing a build from source (or will, once I get the proxy server issues resolved with the build package). Will post an update when I have one.

Having a native client, or at least one written in a high-level language that’s native to Linux, would be nice. Is there a spec for the client API?

[Sebastian Roth]

@mechalas What high-level language have you got in mind? I’d think that re-implementing the whole client is a lot more work than trying to fix work around the certificate store stuff.

[mechalas]

@Sebastian-Roth Probably Python since it has widespread support. Honestly, I don’t even need the full client. I just need the host renamer and the task manager for scheduling reboots. (And I can survive without the latter. I am using Fog in a small lab environment with only a few dozen clients).

I tried a source build of Mono 3.10 which is the version mentioned in the links above but the source build failed on a missing symbol that is internal to the mono package which was not encouraging. Deep dives into mono source builds is not where I want to be spending my time.

Is it possible to just fix the cert store by copying it from a working Ubuntu system? I know very little about mono.

[Sebastian Roth]

@mechalas I can see where you are headed and I do agree that digging much deeper into building Mono from source can be a huge drain of time and effort without knowing the outcome.

There is no full API/call documentation on the client-server communication but the full source code of fog-client and it’s library (called zazzles) is available as reference implementation. But I’d guess it’s just as much a huge effort to get this implemented.

Should we try and focus on fixing the cert store issue? Although it might need a little more effort for you to get into .NET language it’s still way easier than building a new client from scratch I reckon.

I will try to contact the main fog-client developer to get some hints on the exact cert store issue and maybe find some more time to work with you on fixing this. I have done a little bit of fog-client development and know come of the culprits. So I guess we can get to fix this.

[mechalas]

I am an experienced C# developer so .NET isn’t the hangup. I just prefer solutions that are natively supported by the environment they run in. There is less complexity, and less complexity means fewer problems such as this one.

That being said, I’d rather get this working on mono since that’s what’s currently distributed and supported. So, yes, any hints the client developer might have would be greatly appreciated.

[Sebastian Roth]

@mechalas Sorry for the long delay. Took a fair while till I found the time to work on this for a couple of hours straight. After some intense digging I may have found out how to fix the issue, though I am still not absolutely sure why this fails on some Mono installations but works on others and on Windows.

Seems like newer Mono versions do some more validity checking on the certificates that are in the store and don’t return the FOG CA certificate for us. I found a quick way to read the full certificate store and manually match the one we are looking for. That’s not a perfect solution but it’s enough to give it a try and see if I am on the right track with this. I tested this on CentOS 7 and it works with a fixed fog-client library. See below.

Download Zazzles_fixed_Linux.dll, stop FOGService (systemctl stop FOGService), rename /opt/fog-service/Zazzles.dll and put the new Zazzles.dll in place. Then start the FOGService again and watch the log.

Edit: Fixed the path…

[mechalas]

Thanks, @Sebastian-Roth. I will try this out.

[mechalas]

This worked! Thank you so much for the debugging and the resolution.

FOG has been a huge boon to our lab, and getting this last, niggling issue resolved is very much appreciated.

[Sebastian Roth]

@mechalas Thanks for testing and reporting back! Feeling relieved that we have this one nailed down as I had expected it to be a way bigger mess when I started to work on this.

Though still I need to figure out why the normal cert store Find function wouldn’t return the FOG certificates. You said you know C#. Do you think you’d find a bit of time to help investigate on the root cause of this? I can send you a code sample that is very easy to build. It’s help me a lot as I probably won’t get to dig into this again myself in the next couple of days and I’d love to get this finished and properly fixed.

[mechalas]

@Sebastian-Roth I just realized I never responded to this. Ooops!

I would be happy to look at this if you are still up for it.

[Sebastian Roth]

@mechalas Nice to see you’re still around! I remember having looked into this for quite a while but then decided I’d just use the cert store search as implemented to fix this. It’s working great and I don’t see any drawbacks. We have moved forward and released new fog-client versions since then.

Nevertheless there are new things to look at and figure out. It would be really great if you’d help us on the fog-client.