RHEL 8, CentOS 8, master branch, July 22, 2021


  • I’ve been monitoring & have once cleaned up the daily installation tests, and for a while now RHEL 8 and CentOS 8 have been failing every day on the master branch, while the dev branch passes tests every day.

    Given the dev branch is working fine, it would seem the fix is to release the current dev as a new version. What sort of shape is the dev branch concerning all the other aspects of FOG? Is it in a shape to release?

    The external reporting graphs show 50 servers on 1.5.9.98, with overall about 460 servers operating on the dev branch.

    There is another aspect to this - maybe we don’t prioritize a release just for fixing CentOS 8 and RHEL8. It’s a legitimate question to ask, at least. External reporting shows CentOS 8 and CentOS Stream 8 being around 15 total for the dev branch. Pretty low figure. RHEL doesn’t even register in the top 20 OS versions in use. Raspbian has a higher adoption on dev branch than RHEL.

  • Senior Developer

    @sebastian-roth said in RHEL 8, CentOS 8, master branch, July 22, 2021:

    sed -i -e 's/dnf check-update/dnf -y check-update/g' /path/to/fogproject/lib/redhat/config.
    

    Not sure if you saw that I edited my initial post. That sed command should take care of the issue that is causing master to fail. Find the related commit that fixed the issue in dev-branch here: https://github.com/FOGProject/fogproject/commit/90caf2367d21d0594b7f16b2813a2fa94684ad78



  • Today pictured below, for example. This is what normally happens. Master fails and dev-branch succeeds. Yet, dev-branch and master branch are executing against the same OS, same patch level. The way these tests are setup is - the VMs get a snapshot restored. Then they are patched. Then they get the specified branch installed. This is repeated for each branch. Fresh snapshot applied and fresh patching performed before every installation.

    CentOS 8 Output for master branch: 2021-07-30_12-00-PM_output.log

    CentOS 8 Error Log for master branch: 2021-07-30_12-00-PM_fog_error.log

    dev-branch fog_error output for CentOS 8:

    package remi-release-8* is not installed
    warning: /var/tmp/rpm-tmp.AVYE4j: Header V4 RSA/SHA256 Signature, key ID 5f11735a: NOKEY
    Retrieving https://rpms.remirepo.net/enterprise/remi-release-8.rpm
    Verifying...                          ########################################
    Preparing...                          ########################################
    Updating / installing...
    remi-release-8.4-1.el8.remi           ########################################
    Extra Packages for Enterprise Linux Modular 8 - 1.9 MB/s | 798 kB     00:00    
    Extra Packages for Enterprise Linux 8 - x86_64   24 MB/s |  10 MB     00:00    
    Remi's Modular repository for Enterprise Linux  772  B/s | 858  B     00:01    
    Remi's Modular repository for Enterprise Linux  3.0 MB/s | 3.1 kB     00:00    
    Importing GPG key 0x5F11735A:
     Userid     : "Remi's RPM repository <remi@remirepo.net>"
     Fingerprint: 6B38 FEA7 231F 87F5 2B9C A9D8 5550 9759 5F11 735A
     From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el8
    Remi's Modular repository for Enterprise Linux  795 kB/s | 779 kB     00:00    
    Safe Remi's RPM repository for Enterprise Linux 1.6 kB/s | 858  B     00:00    
    Safe Remi's RPM repository for Enterprise Linux 3.0 MB/s | 3.1 kB     00:00    
    Importing GPG key 0x5F11735A:
     Userid     : "Remi's RPM repository <remi@remirepo.net>"
     Fingerprint: 6B38 FEA7 231F 87F5 2B9C A9D8 5550 9759 5F11 735A
     From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el8
    Safe Remi's RPM repository for Enterprise Linux 1.9 MB/s | 1.9 MB     00:01    
    bc-1.07.1-5.el8.x86_64
    curl-7.61.1-18.el8.x86_64
    

    master branch fog_error output for CentOS8:

    package remi-release-8* is not installed
    warning: /var/tmp/rpm-tmp.fFeNbj: Header V4 RSA/SHA256 Signature, key ID 5f11735a: NOKEY
    Retrieving http://rpms.remirepo.net/enterprise/remi-release-8.rpm
    Verifying...                          ########################################
    Preparing...                          ########################################
    Updating / installing...
    remi-release-8.4-1.el8.remi           ########################################
    Extra Packages for Enterprise Linux Modular 8 - 4.7 MB/s | 798 kB     00:00    
    Extra Packages for Enterprise Linux 8 - x86_64   32 MB/s |  10 MB     00:00    
    Remi's Modular repository for Enterprise Linux  568  B/s | 858  B     00:01    
    Remi's Modular repository for Enterprise Linux  3.0 MB/s | 3.1 kB     00:00    
    Importing GPG key 0x5F11735A:
     Userid     : "Remi's RPM repository <remi@remirepo.net>"
     Fingerprint: 6B38 FEA7 231F 87F5 2B9C A9D8 5550 9759 5F11 735A
     From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el8
    Is this ok [y/N]: Remi's Modular repository for Enterprise Linux  2.4 kB/s | 858  B     00:00    
    Error: Failed to download metadata for repo 'remi-modular': repomd.xml GPG signature verification error: Bad GPG signature
    bc-1.07.1-5.el8.x86_64
    curl-7.61.1-18.el8.x86_64
    

    Firefox_Screenshot_2021-07-30T17-39-55.385Z.png


  • @sebastian-roth Yeah, today’s tests look like a mess. Though, what your saying about this GPG key. All the tests use the same OS, same patch level each day. If there was an issue with a GPG key in master branch, wouldn’t it be true that the same problem exists in dev branch? Most days, this is not the case. Most days, just master branch fail for CentOS 8 and RHEL 8.

    There are ways to inject commands before installation, though the point of these tests is just to see in an automatic way if there are any issues that arise for a typical installation.

  • Senior Developer

    @Wayne-Workman Figured that you can’t confirm the GPG key for REMI’s repo when running the installer in silent mode (confirm all questions). We do import the key via rpm --import ... via the installer script but turns out this is not enough when dnf is used because it uses a different gpg database “by design”: https://bugzilla.redhat.com/show_bug.cgi?id=1768206#c4

    Do you have some kind of mechanism in your test setup that allows to run some commands before the installer test? You might run this command on CentOS 8 and RHEL 8 for the time being.

    sed -i -e 's/dnf check-update/dnf -y check-update/g' /path/to/fogproject/lib/redhat/config.sh
    

    As I said, we work on the next release but can’t give you a date yet.

    Looking at the test results today we see that dev-branch fails on CentOS 8 and RHEL 8 as well. Strange error for RHEL 8:

    package remi-release-8* is not installed
    warning: /var/tmp/rpm-tmp.da0Ljt: Header V4 RSA/SHA256 Signature, key ID 5f11735a: NOKEY
    Retrieving https://rpms.remirepo.net/enterprise/remi-release-8.rpm
    Verifying...                          ########################################
    Preparing...                          ########################################
    Updating / installing...
    remi-release-8.4-1.el8.remi           ########################################
    curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to rpms.remirepo.net:443 
    error: https://rpms.remirepo.net/RPM-GPG-KEY-remi: import read failed(2).
    

    And similar for CentOS 8:

    package remi-release-8* is not installed
    curl: (52) Empty reply from server
    error: skipping https://rpms.remirepo.net/enterprise/remi-release-8.rpm - transfer failed
    Retrieving https://rpms.remirepo.net/enterprise/remi-release-8.rpm
    

    And even CentOS 7:

    package remi-release-7* is not installed
    curl: (56) TCP connection reset by peer
    error: skipping https://rpms.remirepo.net/enterprise/remi-release-7.rpm - transfer failed
    Retrieving https://rpms.remirepo.net/enterprise/remi-release-7.rpm
    

    Probably a network issue…

  • Senior Developer

    @Wayne-Workman I am pretty sure I did a plain test install with 1.5.9 on CentOS 8 and it worked. Will try again.


  • @sebastian-roth said in RHEL 8, CentOS 8, master branch, July 22, 2021:

    Seems like the REMI GPG key changed. You need to manually run dnf update once and confirm the GPG key import.
    Interesting this is not happening for the dev-branch tests.

    @Sebastian-Roth I have done this. Which is why I’m bringing it up again.

  • Senior Developer

    @Wayne-Workman Thanks for bringing this up. Before I answer your question on the next release I might quote what I sent you in a private message some days ago:

    @Sebastian-Roth said:

    Seems like the REMI GPG key changed. You need to manually run dnf update once and confirm the GPG key import.
    Interesting this is not happening for the dev-branch tests.

    I still try to get things ready for a very last 1.5.x release. As my time is kind of limited to work on this it’s just taking ages. Still can’t give you a definite date on when it’s all ready. Anyone is welcome to contact me and help getting the last bits done. Although I have to say that good knowledge of FOG is needed to be of help.

    PS: As CentOS is fading we need to look at the successors as well. I did a test install on Rocky 8 as well as AlmaLinux 8 and found both worked. So we are on a good road with those as well.

279
Online

8.8k
Users

15.5k
Topics

144.4k
Posts