FOG client certificates removed on mono update


  • Hello FOG Team,

    I just encountered the problem, that the FOG certificates are removed when mono is updated.
    This is of course horrible, because I need to re-install the FOG client on those machines manually since they lose their connection to the FOG server.

    During installation of certifcates it looks like that:

    Processing triggers for ca-certificates (20210119~20.04.1) ...^M
    Updating certificates in /etc/ssl/certs...^M
    0 added, 0 removed; done.^M
    Running hooks in /etc/ca-certificates/update.d...^M
    ^M
    done.^M
    Updating Mono key store^M
    Mono Certificate Store Sync - version 6.12.0.107^M
    Populate Mono certificate store from a concatenated list of certificates.^M
    Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.^M
    ^M
    Importing into legacy system store:^M
    I already trust 131, your new list has 129^M
    2 previously trusted certificates were removed.^M
    Certificate removed: C=US, S=Illinious, L=Chicago, O=FOG Project, CN=FOG Project CA, E=noreply@fogproject.org^M
    Certificate removed: CN=FOG Server CA^M
    Import process completed.^M
    ^M
    Importing into BTLS system store:^M
    I already trust 129, your new list has 129^M
    Import process completed.^M
    Done^M
    done.
    

    I googled the problem and found this issue:
    https://github.com/mono/mono/issues/9496

    with the recommendation:

    This is expected behaviour, but obviously not desired in this instance.

    I’ll think about how to avoid this problem, without losing the (important) ability of cert-sync to remove no-longer-OS-trusted certificates.

    Workaround for now: trust the cert at the OS level, by saving it in PEM format as /usr/local/share/ca-certificates/foo.crt

    But it seems that the workaround stayed the solution.
    Do you think that the described workaround is the best solution in this case or do you have an alternative idea 🙂

    Best,
    Markus


  • @Sebastian-Roth It did containing

     04/13/2021 11:58:30 Installer Settings successfully saved in /opt/fog-service/settings.json
     04/13/2021 11:58:30 Middleware::Communication Download: http://fog-pxe-srv.XXX.de/fog/management/other/ca.cert.der
    
  • Senior Developer

    @mstabrin Did it create the mentioned log file as well?


  • @Sebastian-Roth i checked the mono certificates and I did not see any fog ones anymore 🙂

    I also checked the other issues, but I could only see beautiful green PASS messages 🙂

  • Senior Developer

    @mstabrin Thanks for testing and reporting back so quickly!

    I would hope you don’t see anything in the certificate store anymore, right?


  • @Sebastian-Roth It even worked with my custom FOGMontior wrapper script and I did not receive any errors 🙂

    Best,
    Markus


  • @Sebastian-Roth Hello, so I tested the installer and did not receive any errors.

  • Senior Developer

    @mstabrin Finally found the time to work on this again. Find an updated SmartInstaller for testing on github: https://github.com/FOGProject/fog-client/releases/download/0.12.0/SmartInstaller_use-cert-from-local-file.exe

    Please let me know if this works as expected and all the things mentioned are fixed now.


  • @Sebastian-Roth No worries, take your time 🙂 Right now, most mono updates happen during a snapin deployment and as a workaround I copy the needed certificates back to the mono directory at the end of the snapin.
    While this is not ideal, it should not happen too often within the next weeks and therefore is not a too pressing issue right now 🙂

  • Senior Developer

    @mstabrin I did not find enough time to further work on this topic, I am sorry. Will try to next week, though I can’t promise I will get to it.

  • Senior Developer

    @mstabrin Thanks for testing and letting me know. I will look into this the next days again.


  • @Sebastian-Roth So I gave it a try and things appear to be working in some way!

    Installation looks like that

    ----------------------------------Information---------------------------------
    
    Version.................................................................0.12.1
    OS.......................................................................Linux
    Current Path....................................................../home/cadmin
    Install Location............................................../opt/fog-service
    Systemd...................................................................True
    Initd.....................................................................True
    
    -----------------------------------Configure----------------------------------
    
    FOG Server address [default: fogserver]: FOG-PXE-SRV.mpi-dortmund.mpg.de
    Webroot [default: /fog]:                 
    Enable tray icon? [Y/n]:                 
    Start FOG Service when done? [Y/n]:      
    
    ----------------------------------Installing----------------------------------
    
    Getting things ready....................................................[Pass]
    Installing files........................................................[Pass]
    Saving Configuration.................................................... 03/22/2021 08:08:15 Installer Settings successfully saved in /opt/fog-service/settings.json
    [Pass]
    Applying Configuration..................................................[Pass]
    Pinning FOG Project..................................................... 03/22/2021 08:08:15 Installer FOG Project CA successfully installed
    [Pass]
    Pinning Server.......................................................... 03/22/2021 08:08:15 Data::RSA Unable to use CA cert from /home/cadmin/ca.cert.der, trying cert store now.
     03/22/2021 08:08:15 Data::RSA ERROR: FOG Server CA NOT found in keystore - needs to be installed
     03/22/2021 08:08:15 Middleware::Communication Download: http://FOG-PXE-SRV.mpi-dortmund.mpg.de/fog/management/other/ca.cert.der
     03/22/2021 08:08:16 Installer Successfully pinned server CA cert to CN=FOG Server CA
    [Pass]
    
    Starting FOG Service....................................................[Pass]
    
    -----------------------------------Finished-----------------------------------
    
    See /home/cadmin/SmartInstaller.log for more information.
    

    However, this line is huge and red in my terminal and can be a disturbing factor for some people I suppose:
    03/22/2021 08:08:15 Data::RSA ERROR: FOG Server CA NOT found in keystore - needs to be installed
    Additionally it also does not go away when you install the program a second time on top of the first, so I assume that the keystore is still searched even though it is no longer used 🙂

    This line is also confusing: See /home/cadmin/SmartInstaller.log for more information.
    Because there is no log file created by the installer in the first place (I checked the previous installer 1.12.0 and there also no log file was created).

    Prior installation I deleted all the mono certificates that contained FOG, but after the installation I checked the mono certificates and it appears that the tbp file found its way back:

    root@pcf-server2021:/home/cadmin# grep FOG /usr/share/.mono/certs/Trust/* Binary file /usr/share/.mono/certs/Trust/tbp-090753F074AB4FB3C022CCC655B02AD21436BAD5E1191CF8870273478E46438D.cer matches

    After a encrypten reset, thinks appear to be working though.
    I will keep you updated!


  • @Sebastian-Roth Alright!
    I will give it a try now 🙂

  • Senior Developer

    @mstabrin Ok, I got a first test build ready for you. As far as I see it’s all in the Zazzles.dll library we provide and so you should be able to just swap that out and give it a try.

    Get that machine ready you posted the “mono removed certs” log from. Download the modfied DLL and rename to Zazzles.dll. Stop fog-client (systemctl status FOGService) and rename /opt/fog-service/Zazzles.dll (e.g. Zazzles.dll.orig). Put that new DLL in place of the old one and start fog-client.

    If you run into any issue, please provide log information! If everything goes well you should now see something like “Using FOG Server CA from file …” in fog.log.

    If that works you might go ahead for the next test. Download the modified Installer (non-official pre 0.12.1 version) and install that on any machine you like. Try one that has not had fog-client yet or one where 0.12.0 was installed before. That will install without installing the FOG Server CA cert into the mono key store. It should work just as well. I mean communicating to your FOG server using the local CA cert file to verify communication instead of trying to draw it from mono’s keystore.


  • @Sebastian-Roth Sure I will be around and I will see how I can stop the update from happening

    Looking forward to it!

    Best,
    Markus

  • Senior Developer

    @mstabrin It’s probably time to finally replace the mono certificate store and use a simple cert file in Linux/MacOSX and maybe even Windows. We’ve had this recorded on GitHub for a long time but never found the time to work on this.

    https://github.com/FOGProject/zazzles/issues/23

    I remember someone posting about a slightly different issue with mono cert store not long ago. So it would make sense to get into this finally.

    Will you be around the next weeks and have time to test?

    Can you hold back on updating mono until we have a fog-client ready for auto-updating so you don’t need to re-install all of them manually?

340
Online

8.1k
Users

15.0k
Topics

141.4k
Posts