FOG-Client suddenly stopped working

Sebastian Roth

@kek It’s not even been a whole year but I did not remember we had this error reported before - thanks to the memory of the fourms. It looks like there is an issue in the way Linux Mono is reading the certifcates from the store causing it to fail when matching the CA certificate to the one it loads from your FOG server.

It’s strange/interesting you get this error just now. Possibly a Mono update on your Linux systems??

Try the following fix: Download Zazzles.dll and put in /opt/fog-service/Zazzles.dll (rename the original one). Then stop and restart the fog-client or reboot the computer and check the logs.

We have about 100 Linux Clients, is there a server solution without touching them all?

Maybe there is. We could come up with a so called “post init” script that could deploy that DLL to selected hosts e.g. via a host inventory task. The hosts would need to PXE boot once to get this done.

Or I might suggest you use clusterssh to install that fix to all your hosts in batches of 10 or more (depending on the size of screen you have).

I would say test this first on a couple of hosts if it is actually fixing the issue for you and we’ll take it from there.

kek

@sebastian-roth said in FOG-Client suddenly stopped working:

It’s strange/interesting you get this error just now. Possibly a Mono update on your Linux systems??

No we use a fixed Version of Mono (Mono Repository with specific Version specified), but can’t tell you at the moment what version we are using exactly. We did no Updates on both Server and Client.

Maybe we can find the Problem, with the new Information provided. I will try to manually replace the Zazzles.dll on one Client just to see if it works. We also have an Internal Repository, we could update the Mono-Package and create a package for the FOG-Client. But because we Updating our Clients also via FOG-Client we still need to touch them all, but with this solution we have at least a GUI to Update.

kek

I re-checked with my colleagues, not all clients have this Problem, just a few servers, so no problem to replace the Zazzles.dll, but with HTTPS, 1 in settings.json, (and newest 0.12.0 Version fog-client) we get:

------------------------------------------------------------------------------
--------------------------------Authentication--------------------------------
------------------------------------------------------------------------------
 1/11/2021 12:34:34 PM Client-Info Version: 0.12.0
 1/11/2021 12:34:34 PM Client-Info OS:      Linux
 1/11/2021 12:34:34 PM Middleware::Authentication Waiting for authentication timeout to pass
 1/11/2021 12:34:34 PM Middleware::Communication Download: https://<fogdomain>/fog/management/other/ssl/srvpublic.crt
 1/11/2021 12:34:36 PM Data::RSA FOG Server CA cert found
 1/11/2021 12:34:36 PM Data::RSA ERROR: Certificate validation failed
 1/11/2021 12:34:36 PM Data::RSA ERROR: Trust chain did not complete to the known authority anchor. Errors: PartialChain (PartialChain)
 1/11/2021 12:34:36 PM Middleware::Communication SSL certificate chain error: NotTimeValid
 1/11/2021 12:34:36 PM Middleware::Communication ERROR: Could not download file
 1/11/2021 12:34:36 PM Middleware::Communication ERROR: Error: TrustFailure (Authentication failed, see inner exception.)

Sebastian Roth

@kek said in FOG-Client suddenly stopped working:

Middleware::Communication SSL certificate chain error: NotTimeValid

Seems like you have a different issue here. Is the srvpublic.crt still valid?

kek

@sebastian-roth said in FOG-Client suddenly stopped working:

Is the srvpublic.crt still valid?

Was the first thought, but no, its valid.

Sorry for the late answer, very busy at the moment.

Sebastian Roth

@kek I have that feeling that it might be the CA certificate not being valid anymore. On install the fog-client software grabs that CA cert from your FOG server and installs it into mono’s certificate store.

Run certmgr -list -c -v -m Trust as root to see if a CA cert named FOG Server CA is there and still valid.

kek

@sebastian-roth said in FOG-Client suddenly stopped working:

Run certmgr -list -c -v -m Trust as root to see if a CA cert named FOG Server CA is there and still valid.

Output (end):

Unhandled Exception:
System.Security.Cryptography.CryptographicException: Unsupported hash algorithm: 1.2.840.10045.4.3.3
  at Mono.Security.X509.X509Certificate.get_Signature () [0x001a1] in <c8dae181eb1743bd94c3ab5b607caeb0>:0
  at Mono.Tools.CertificateManager.DisplayCertificate (Mono.Security.X509.X509Certificate x509, System.Boolean machine, System.Boolean verbose) [0x00132] in <52d071828ee44ab9ab181a6c5989b2db>:0
  at Mono.Tools.CertificateManager.List (Mono.Tools.CertificateManager+ObjectType type, Mono.Security.X509.X509Store store, System.Boolean machine, System.String file, System.Boolean verbose) [0x0002b] in <52d071828ee44ab9ab181a6c5989b2db>:0
  at Mono.Tools.CertificateManager.Main (System.String[] args) [0x00204] in <52d071828ee44ab9ab181a6c5989b2db>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.Security.Cryptography.CryptographicException: Unsupported hash algorithm: 1.2.840.10045.4.3.3
  at Mono.Security.X509.X509Certificate.get_Signature () [0x001a1] in <c8dae181eb1743bd94c3ab5b607caeb0>:0
  at Mono.Tools.CertificateManager.DisplayCertificate (Mono.Security.X509.X509Certificate x509, System.Boolean machine, System.Boolean verbose) [0x00132] in <52d071828ee44ab9ab181a6c5989b2db>:0
  at Mono.Tools.CertificateManager.List (Mono.Tools.CertificateManager+ObjectType type, Mono.Security.X509.X509Store store, System.Boolean machine, System.String file, System.Boolean verbose) [0x0002b] in <52d071828ee44ab9ab181a6c5989b2db>:0
  at Mono.Tools.CertificateManager.Main (System.String[] args) [0x00204] in <52d071828ee44ab9ab181a6c5989b2db>:0

Sebastian Roth

@kek said in FOG-Client suddenly stopped working:

CryptographicException: Unsupported hash algorithm: 1.2.840.10045.4.3.3

Do you see any certificate information before the exception happens? Searching the web for this message I found some people reporting the same - very old and newer information as well:

Which version of mono do you use?

I still can’t get my head around why this used to work but now seems to fail so badly. Have you tried installing the fog-client on a fresh new system? As well, what happens if you remove and re-install the fog-client (and mono?) on a system?

To me it seems like something might have corrupted the mono certificate store. See if you can find that store in /usr/share/.mono/certs/?! What is the last change date of the files in that directory?

kek

UPDATE: followed the tutorial at: https://wiki.fogproject.org/wiki/index.php/FOG_Client#Installing_-_Linux
And have now mono-complete Version 6.12.0.107-0xamarin13+debian10b1
But same messages in the log…

Sorry for the late reply, we are very busy at the moment…

So we reinstalled our FOG-Server last week, and now the Server and the Storage-Nodes are up-to-date (1.5.9) from the old Server we copied over /var/www/fog/management/other/ssl/srvpublic.crt, and we have as mentioned earlier no problems with most of the Clients. Only eight of our Servers (total), still have problems… So with Server-Version 1.5.9 and Client version 0.12.0 we still have problems, here the log:

 2/19/2021 5:18:13 PM Main Overriding exception handling
 2/19/2021 5:18:13 PM Main Bootstrapping Zazzles
 2/19/2021 5:18:13 PM Controller Initialize
 2/19/2021 5:18:13 PM Controller Start

 2/19/2021 5:18:13 PM Service Starting service
 2/19/2021 5:18:13 PM Bus Became bus server
 2/19/2021 5:18:13 PM Bus Emmiting message on channel: Status
 2/19/2021 5:18:13 PM Service Invoking early JIT compilation on needed binaries

------------------------------------------------------------------------------
--------------------------------Authentication--------------------------------
------------------------------------------------------------------------------
 2/19/2021 5:18:14 PM Client-Info Version: 0.12.0
 2/19/2021 5:18:14 PM Client-Info OS:      Linux
 2/19/2021 5:18:14 PM Middleware::Authentication Waiting for authentication timeout to pass
 2/19/2021 5:18:14 PM Middleware::Communication Download: http://<fogserver>/fog/management/other/ssl/srvpublic.crt
 2/19/2021 5:18:14 PM Data::RSA FOG Server CA cert found
 2/19/2021 5:18:14 PM Data::RSA ERROR: Certificate validation failed
 2/19/2021 5:18:14 PM Data::RSA ERROR: Trust chain did not complete to the known authority anchor. Errors: NotSignatureValid (NotSignatureValid)
 2/19/2021 5:18:14 PM Middleware::Authentication ERROR: Could not authenticate
 2/19/2021 5:18:14 PM Middleware::Authentication ERROR: Certificate is not from FOG CA

------------------------------------------------------------------------------
--------------------------------Authentication--------------------------------
------------------------------------------------------------------------------
 2/19/2021 5:18:14 PM Client-Info Version: 0.12.0
 2/19/2021 5:18:14 PM Client-Info OS:      Linux
 2/19/2021 5:18:14 PM Middleware::Authentication Waiting for authentication timeout to pass

Complete output of certmgr -list -c -v -m Trust:

Mono Certificate Manager - version 5.18.0.240
Manage X.509 certificates and CRL from stores.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Self-signed X.509 v3 Certificate
  Serial Number: 4AC79159C96A75A1B146429056E03B08
  Issuer Name:   C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA
  Subject Name:  C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA
  Valid From:    11/10/2006 12:00:00 AM
  Valid Until:   11/10/2031 12:00:00 AM
  Unique Hash:   B34DDD372ED92E8F2ABFBB9E20A9D31F204F194B
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:         (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 2D99D41C39044F7C
  Issuer Name:   C=US, O=AffirmTrust, CN=AffirmTrust Networking
  Subject Name:  C=US, O=AffirmTrust, CN=AffirmTrust Networking
  Valid From:    1/29/2010 2:08:24 PM
  Valid Until:   12/31/2030 2:08:24 PM
  Unique Hash:   2110A6E8DA67CEE9D90CCBF913117C60EC31C914
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: C04404
  Issuer Name:   C=PL, O=Unizeto Technologies S.A., OU=Certum Certification Authority, CN=Certum Trusted Network CA
  Subject Name:  C=PL, O=Unizeto Technologies S.A., OU=Certum Certification Authority, CN=Certum Trusted Network CA
  Valid From:    10/22/2008 12:07:37 PM
  Valid Until:   12/31/2029 12:07:37 PM
  Unique Hash:   A8569CCD21EF9CC5737C7A12DF608C2CBC545DF1
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 00
  Issuer Name:   C=JP, O="SECOM Trust Systems CO.,LTD.", OU=Security Communication RootCA2
  Subject Name:  C=JP, O="SECOM Trust Systems CO.,LTD.", OU=Security Communication RootCA2
  Valid From:    5/29/2009 5:00:39 AM
  Valid Until:   5/29/2029 5:00:39 AM
  Unique Hash:   453ECC5C2C07CCC737ABCA4F06054723F20169FCE993F86657343DB97515C000
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.11
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 3DE54602353EEE020BE065828A2D814E
  Issuer Name:   C=GB, S=Greater Manchester, L=Salford, O=COMODO CA Limited, CN=COMODO Certification Authority
  Subject Name:  C=GB, S=Greater Manchester, L=Salford, O=COMODO CA Limited, CN=COMODO Certification Authority
  Valid From:    12/1/2006 12:00:00 AM
  Valid Until:   12/31/2029 11:59:59 PM
  Unique Hash:   C1F49DACC04C76C9D07297565C4C2FDA367B90DC
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 4AE671E3D889CA4C003FED73A0F98054
  Issuer Name:   C=EE, O=AS Sertifitseerimiskeskus, CN=EE Certification Centre Root CA, E=pki@sk.ee
  Subject Name:  C=EE, O=AS Sertifitseerimiskeskus, CN=EE Certification Centre Root CA, E=pki@sk.ee
  Valid From:    10/30/2010 10:10:30 AM
  Valid Until:   12/17/2030 11:59:59 PM
  Unique Hash:   3FD9A3751E2081CB6BF65CCEBD588623D20D9A61
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: A45A1CB823AEC6C4DF4093C900ECA54C8A5F1608
  Issuer Name:   C=HK, S=Hong Kong, L=Hong Kong, O=Hongkong Post, CN=Hongkong Post Root CA 3
  Subject Name:  C=HK, S=Hong Kong, L=Hong Kong, O=Hongkong Post, CN=Hongkong Post Root CA 3
  Valid From:    6/3/2017 2:29:46 AM
  Valid Until:   6/3/2042 2:29:46 AM
  Unique Hash:   D6ED17A5F51972C262E2D3A8677577857C6A85700A2D22E0A4F87948D6834F63
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.11
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 4B2FBB542FD41B4F
  Issuer Name:   C=CH, O=SwissSign AG, CN=SwissSign Silver CA - G2
  Subject Name:  C=CH, O=SwissSign AG, CN=SwissSign Silver CA - G2
  Valid From:    10/25/2006 8:32:46 AM
  Valid Until:   10/25/2036 8:32:46 AM
  Unique Hash:   526AAA5D52A07C057AD6E17522FB678A3E154558
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: E1A6E3C46D41E6A30D0355F1891BE9CA00
  Issuer Name:   C=FR, O=Dhimyotis, OU=0002 48146308100036, CN=Certigna Root CA
  Subject Name:  C=FR, O=Dhimyotis, OU=0002 48146308100036, CN=Certigna Root CA
  Valid From:    10/1/2013 8:32:27 AM
  Valid Until:   10/1/2033 8:32:27 AM
  Unique Hash:   9668D6C44B5F62EE4A56423640D93D45A2C772C6D42ED178978AF5ADDB15FDAE
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.11
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 0905
  Issuer Name:   C=BM, O=QuoVadis Limited, CN=QuoVadis Root CA 2
  Subject Name:  C=BM, O=QuoVadis Limited, CN=QuoVadis Root CA 2
  Valid From:    11/24/2006 6:27:00 PM
  Valid Until:   11/24/2031 6:23:33 PM
  Unique Hash:   C8F8A3C6BF401D34E6F1D8F8E1DDD08BBB934626
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: FF48C90F01E3DCFE00
  Issuer Name:   C=FR, O=Dhimyotis, CN=Certigna
  Subject Name:  C=FR, O=Dhimyotis, CN=Certigna
  Valid From:    6/29/2007 3:13:05 PM
  Valid Until:   6/29/2027 3:13:05 PM
  Unique Hash:   D49BA8CA0DB5E6C661B57B56F33B4F05163FF8F2
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 7DE619D78BF5CBE1BF5F48165AB7B000
  Issuer Name:   C=ES, O=IZENPE S.A., CN=Izenpe.com
  Subject Name:  C=ES, O=IZENPE S.A., CN=Izenpe.com
  Valid From:    12/13/2007 1:08:28 PM
  Valid Until:   12/13/2037 8:27:25 AM
  Unique Hash:   9E5428441BEFFA8BCFD95D3272309D63A6AB83812A09D6D7A71B514408AF47A1
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.11
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 9DBCD206E45E0097B8AF5C4765BDC815
  Issuer Name:   C=TW, O="Chunghwa Telecom Co., Ltd.", OU=ePKI Root Certification Authority
  Subject Name:  C=TW, O="Chunghwa Telecom Co., Ltd.", OU=ePKI Root Certification Authority
  Valid From:    12/20/2004 2:31:27 AM
  Valid Until:   12/20/2034 2:31:27 AM
  Unique Hash:   E2D1E7E0391A13E13A9759961938A4FAAB8DEA65
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: E0683190E3171647E6165CC26F33CB57
  Issuer Name:   C=US, O=Network Solutions L.L.C., CN=Network Solutions Certificate Authority
  Subject Name:  C=US, O=Network Solutions L.L.C., CN=Network Solutions Certificate Authority
  Valid From:    12/1/2006 12:00:00 AM
  Valid Until:   12/31/2029 11:59:59 PM
  Unique Hash:   930DBFC5830B7BFD486F9056FCB8751F3D21BF12
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           3082010A0282010100E4BC7E92306DC6D88E2B0BBC46CEE02796DEDEF9FA12D33C3373B3042FBC718CE59FB622603E5F5DCE09FF820C1B9A51501A2689DDD5615D19DC120F2D0AA2435D17D0349220EA73CF382C0626097A72F7FA5032F8C293D369A223CE41B1CCE4D51F36D18A3AF88C63E2145969ED0DD37F6BE8B803E54F6AE59863694805BE2EFF33B6E9975969F86719AE9361964415D372B03FBC6A7DEC487F8DC3ABAA712B5369415334B5B0B9C5060AC4B045F5415D6E89457B3D3B268C74C2E5D2D17DB211D4FB5832229A80C9DCFD0CE97F5E0397CE3B001487277038A98E6EB327769851E005E321AB1AD585223C29B59A16C580A8F4BB6B308F2F4602A2B10C22E0D30203010001
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            BBAE4BE7B757EB7FAA2DB77347856AC1E4A51DE4E73CE9F4596577B57A5B5A8D2536E07A972E38C05760839806839FB9767A6E50E0BA882CFC45CC18B09995510EEC1DB888FF87501C82C2E3E03280BFA00B47C8C331EF996732804F1721790C695CDE5E34AE02B526EA50DF7F18652CC9F263E1A907FE7C711F6B33246A1E05F70568C06A12CB2E5E61CBAE28D37EC2B46691265F3C2E245FCB580FEB28ECAF1196F3DC7B6FC0A788F25377B3605EAEAE28DA352C6F3445D326E1DEEC5B4F276B167CBD44041882B389791710713D7AA2164EF501CDA46C6568A149765C43C9D8BC36676CA594B5D4CCB9BD6A355621DED8C3EBFBCBA4604CB055A0A07B57B2
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: E4D6E4DC2DEB015FB6E3B7D532D255EC075D8A3E
  Issuer Name:   C=PL, O=Krajowa Izba Rozliczeniowa S.A., CN=SZAFIR ROOT CA2
  Subject Name:  C=PL, O=Krajowa Izba Rozliczeniowa S.A., CN=SZAFIR ROOT CA2
  Valid From:    10/19/2015 7:43:30 AM
  Valid Until:   10/19/2035 7:43:30 AM
  Unique Hash:   8F65AB514D193E1BC2C69D82520F73C4E3255744356064E9859107F26C0EFD5C
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           3082010A0282010100B7BC3E50A84BCD40B5CE61E796CAB4A1DA0C22B0FAB57B7600778C0BCF7DA886CC2651E4203D850CD658E3E7F42A189DDAD1AE26EEEB53DCF490D6134A0C903CC3F4DAD28E0D923ADCB1B1FF38DEC3BA2D5F80B902BD4A9D1B0FB4C3C2C16703DDDC1B9C3DB3B0DE001EA83447BB9AEBFE0B14BD3684DA0D20BFFA5BCBA91620AD3960EE2F75B6E7979CF93EFD7E4D6F4D2FEF880D6AFADDF13D6E20A5A012B44D70B9CED7723B8993A780841C27497249B5FF3B959EC1CCC801ECE80E8A0A96E7B3A687E5D6F9052B0D9740703CBAAC755A9CD54D9D020AD24B9B664B46071765AD9F6C8800DC2289E0E164D467BC3179613CBBCA41CD5C6A00C83C388E58AF0203010001
  Signature Algorithm:  1.2.840.113549.1.1.11
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 53CB9B519C3E686A
  Issuer Name:   C=TR, L=Ankara, O=E-Tuğra EBG Bilişim Teknolojileri ve Hizmetleri A.Ş., OU=E-Tugra Sertifikasyon Merkezi, CN=E-Tugra Certification Authority
  Subject Name:  C=TR, L=Ankara, O=E-Tuğra EBG Bilişim Teknolojileri ve Hizmetleri A.Ş., OU=E-Tugra Sertifikasyon Merkezi, CN=E-Tugra Certification Authority
  Valid From:    3/5/2013 12:09:48 PM
  Valid Until:   3/3/2023 12:09:48 PM
  Unique Hash:   AE284D570FF1601F3D9E2067F8B5D44E58B49D5142A2D888235926E44B49A1EB
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.11
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: C54AEFA1421099D600
  Issuer Name:   C=US, S=Illinious, L=Chicago, O=FOG Project, CN=FOG Project, E=noreply@fogproject.org
  Subject Name:  C=US, S=Illinious, L=Chicago, O=FOG Project, CN=FOG Project, E=noreply@fogproject.org
  Valid From:    9/9/2015 1:04:11 AM
  Valid Until:   9/7/2020 1:04:11 AM
  Unique Hash:   A2401FF1B2C3528B250FBA08FEF97C19E570D35C
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.5
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: C0C2F61A23F8B3468785F0745220B176
  Issuer Name:   C=CH, O=WISeKey, OU=OISTE Foundation Endorsed, CN=OISTE WISeKey Global Root GB CA
  Subject Name:  C=CH, O=WISeKey, OU=OISTE Foundation Endorsed, CN=OISTE WISeKey Global Root GB CA
  Valid From:    12/1/2014 3:00:32 PM
  Valid Until:   12/1/2039 3:10:31 PM
  Unique Hash:   04524E82755B1E36393B942C01DEE51978C032D7D4519F7DA6C964ABF89C5EA9
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.11
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

Self-signed X.509 v3 Certificate
  Serial Number: 00
  Issuer Name:   C=US, S=Arizona, L=Scottsdale, O="GoDaddy.com, Inc.", CN=Go Daddy Root Certificate Authority - G2
  Subject Name:  C=US, S=Arizona, L=Scottsdale, O="GoDaddy.com, Inc.", CN=Go Daddy Root Certificate Authority - G2
  Valid From:    9/1/2009 12:00:00 AM
  Valid Until:   12/31/2037 11:59:59 PM
  Unique Hash:   3560E45B41E46B8F36537025D1D5BC02D9652A10645B0EFF69E8B6A52191F335
  Key Algorithm:        1.2.840.113549.1.1.1
  Algorithm Parameters: 0500
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.113549.1.1.11
  Algorithm Parameters: 0500
  Signature:            (removed)
  Private Key:                  False
  KeyPair Key:                  False

X.509 v3 Certificate
  Serial Number: 26CC8089CDDE5671D2C5945AC5998B5C
  Issuer Name:   C=US, S=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust ECC Certification Authority
  Subject Name:  C=US, S=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust ECC Certification Authority
  Valid From:    2/1/2010 12:00:00 AM
  Valid Until:   1/18/2038 11:59:59 PM
  Unique Hash:
  Key Algorithm:        1.2.840.10045.2.1
  Algorithm Parameters: 06052B81040022
  Public Key:           (removed)
  Signature Algorithm:  1.2.840.10045.4.3.3
  Algorithm Parameters: None

Unhandled Exception:
System.Security.Cryptography.CryptographicException: Unsupported hash algorithm: 1.2.840.10045.4.3.3
  at Mono.Security.X509.X509Certificate.get_Signature () [0x001a1] in <c8dae181eb1743bd94c3ab5b607caeb0>:0
  at Mono.Tools.CertificateManager.DisplayCertificate (Mono.Security.X509.X509Certificate x509, System.Boolean machine, System.Boolean verbose) [0x00132] in <52d071828ee44ab9ab181a6c5989b2db>:0
  at Mono.Tools.CertificateManager.List (Mono.Tools.CertificateManager+ObjectType type, Mono.Security.X509.X509Store store, System.Boolean machine, System.String file, System.Boolean verbose) [0x0002b] in <52d071828ee44ab9ab181a6c5989b2db>:0
  at Mono.Tools.CertificateManager.Main (System.String[] args) [0x00204] in <52d071828ee44ab9ab181a6c5989b2db>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.Security.Cryptography.CryptographicException: Unsupported hash algorithm: 1.2.840.10045.4.3.3
  at Mono.Security.X509.X509Certificate.get_Signature () [0x001a1] in <c8dae181eb1743bd94c3ab5b607caeb0>:0
  at Mono.Tools.CertificateManager.DisplayCertificate (Mono.Security.X509.X509Certificate x509, System.Boolean machine, System.Boolean verbose) [0x00132] in <52d071828ee44ab9ab181a6c5989b2db>:0
  at Mono.Tools.CertificateManager.List (Mono.Tools.CertificateManager+ObjectType type, Mono.Security.X509.X509Store store, System.Boolean machine, System.String file, System.Boolean verbose) [0x0002b] in <52d071828ee44ab9ab181a6c5989b2db>:0
  at Mono.Tools.CertificateManager.Main (System.String[] args) [0x00204] in <52d071828ee44ab9ab181a6c5989b2db>:0

mono-complete Version: 5.18.0.240+dfsg-3

Strange thing: Same server (from a software perspective) with all versions the same, it´s working fine…

Mono certificate store: (/usr/share/.mono/certs/) :

ls -lah /usr/share/.mono/certs/
drwxr-xr-x 3 root root 4.0K Feb 19 14:17 .
drwxr-xr-x 5 root root 4.0K Feb 19 14:20 ..
drwxr-xr-x 2 root root  20K Feb 19 17:18 Trust

Sebastian Roth

@kek said in FOG-Client suddenly stopped working:

Self-signed X.509 v3 Certificate
Serial Number: C54AEFA1421099D600
Issuer Name: C=US, S=Illinious, L=Chicago, O=FOG Project, CN=FOG Project, E=noreply@fogproject.org
Subject Name: C=US, S=Illinious, L=Chicago, O=FOG Project, CN=FOG Project, E=noreply@fogproject.org
Valid From: 9/9/2015 1:04:11 AM
Valid Until: 9/7/2020 1:04:11 AM
Unique Hash: A2401FF1B2C3528B250FBA08FEF97C19E570D35C
Key Algorithm: 1.2.840.113549.1.1.1
Algorithm Parameters: 0500
Public Key: (removed)
Signature Algorithm: 1.2.840.113549.1.1.5
Algorithm Parameters: 0500
Signature: (removed)
Private Key: False
KeyPair Key: False

The Valid Until: 9/7/2020 1:04:11 AM portion tells me this is not a properly installed fog-client 0.12.0 as it would install the more recent “FOG Project CA” being valid for way longer! As we see you have 0.12.0 from the logs I would think it just can’t properly install the certificates to the key store.

Strange thing: Same server (from a software perspective) with all versions the same, it´s working fine…

What do you mean by that? This particular client you posted the information here can communicate with a different FOG server just fine?

You need to know that on installation the fog-client will download the specific server CA cert (http://…/fog/management/other/ca.cert.der) and pinn that client to this server. So copying /var/www/fog/management/other/ssl/srvpublic.crt to a different server is not enough!

kek

@sebastian-roth said in FOG-Client suddenly stopped working:

What do you mean by that? This particular client you posted the information here can communicate with a different FOG server just fine?

Update:

We are updating also the storage nodes after the master-server. The problem seems to come from the update, we also updated this storage node that worked before, and now we have the same problem… But the mono-complete Version seems not to change… so it must be another packet… (Both the Server that has a not working Client and the Server that had a working Client before are Storage nodes)

But the Server that had before the Update a working client has Version 0.11.18 installed, and the log is different:

------------------------------------------------------------------------------
--------------------------------Authentication--------------------------------
------------------------------------------------------------------------------
 2/19/2021 9:26:03 PM Client-Info Version: 0.11.18
 2/19/2021 9:26:03 PM Client-Info OS:      Linux
 2/19/2021 9:26:03 PM Middleware::Authentication Waiting for authentication timeout to pass
 2/19/2021 9:28:03 PM Middleware::Communication Download: http://<fogserver>/fog/management/other/ssl/srvpublic.crt
 2/19/2021 9:28:03 PM Data::RSA ERROR: FOG Server CA NOT found in keystore - needs to be installed
 2/19/2021 9:28:03 PM Middleware::Authentication ERROR: Could not authenticate
 2/19/2021 9:28:03 PM Middleware::Authentication ERROR: Value cannot be null.
Parameter name: authority

Sebastian Roth

@kek said in FOG-Client suddenly stopped working:

RSA ERROR: FOG Server CA NOT found in keystore - needs to be installed

I still think someting is messing up your certificate store.

kek

@sebastian-roth said in FOG-Client suddenly stopped working:

You need to know that on installation the fog-client will download the specific server CA cert (http://…/fog/management/other/ca.cert.der) and pinn that client to this server. So copying /var/www/fog/management/other/ssl/srvpublic.crt to a different server is not enough!

SOLVED!

Copied /fog/management/other/ca.cert.der from old to new server, and it works! Also works with HTTPS: 1 in /opt/fog-service/settings.json! I just need to bring all the clients on the Server to version 0.12.0 ( only 8 ). So solution is update/recreate the Server, copy over the all the certs from the old server and install newest FOG-Client on all Hosts.

Can be marked as solved!

Sebastian Roth

@kek said in FOG-Client suddenly stopped working:

install newest FOG-Client on all Hosts.

Unless there is really something strange going on with the CA and certificate generation on the new server I can’t see why you would need to copy the certs from the old server to the new one when you actually run the fog-client installer to the hosts anyway. That would pull down and pin to the (CA) cert of the new server.

Ok, not that I write this I could see what I might have missed so far. Did you do a fresh install of the fog-client on those hosts before? Because when you initially said “FOG-Client suddenly stopped working this Year. Nothing was changed.” I expected this to be hosts with already installed fog-client that stopped to talk to the FOG server. But I might have misunderstood this point?!

So are you saying a fresh install of the fog-client on Linux is not able to communicate with an up to date FOG server?

Which OS and version exactly do you use on the hosts? Sounds like Debian Buster but I want to make sure I can setup the same scenario that you have. As well please let me know which OS and version you use on the server. I will see if I can replicate the issue and think about how to fix this.

The keystore has caused us trouble in the past but it seemed to work fine when I looked at it more than a year ago. Now if it does cause problems again we might think about adding an alternative to the keystore finally: https://github.com/FOGProject/zazzles/issues/23

kek

@sebastian-roth said in FOG-Client suddenly stopped working:

Unless there is really something strange going on with the CA and certificate generation on the new server I can’t see why you would need to copy the certs from the old server to the new one when you actually run the fog-client installer to the hosts anyway.

I think there is something strange going on, it only worked with the old certificate, also don‘t know why, reinstalled client and mono and also deleted all residual folders after uninstallation of mono.
But it only happened after Storage Node update (1.5.8 to 1.5.9). (We have 8 linux servers, 3 of them are Storage nodes. All have Debian Buster OS)
But most of our linux machines with client (~150) had never problems, but they use all a old fixed version of mono, because they are Lubuntu 18.04 LTS OS and the Ubuntu repo has no mono included (They are also using some old version of the Client (0.11.1x). Master-Server is also Debian 10.

So are you saying a fresh install of the fog-client on Linux is not able to communicate with an up to date FOG server?

Yes. Unless you copy over the cert from the old server. (ca.cert.der)

Sebastian Roth

@kek said in FOG-Client suddenly stopped working:

but they use all a old fixed version of mono, because they are Lubuntu 18.04 LTS OS and the Ubuntu repo has no mono included

That is 5.18.0.240+dfsg-3 as you posted earlier??

Sebastian Roth

@kek Just a quick update as I got to work allowing the fog-client to use a simple local CA certificate file instead of the mono keystore stuff. Would you be interested to test?

FOG-Client suddenly stopped working

196

11.6k

17.1k

154.6k