RHEL 8, CentOS 8, master branch, July 22, 2021

Wayne Workman

I’ve been monitoring & have once cleaned up the daily installation tests, and for a while now RHEL 8 and CentOS 8 have been failing every day on the master branch, while the dev branch passes tests every day.

Given the dev branch is working fine, it would seem the fix is to release the current dev as a new version. What sort of shape is the dev branch concerning all the other aspects of FOG? Is it in a shape to release?

The external reporting graphs show 50 servers on 1.5.9.98, with overall about 460 servers operating on the dev branch.

There is another aspect to this - maybe we don’t prioritize a release just for fixing CentOS 8 and RHEL8. It’s a legitimate question to ask, at least. External reporting shows CentOS 8 and CentOS Stream 8 being around 15 total for the dev branch. Pretty low figure. RHEL doesn’t even register in the top 20 OS versions in use. Raspbian has a higher adoption on dev branch than RHEL.

Sebastian Roth

@Wayne-Workman Thanks for bringing this up. Before I answer your question on the next release I might quote what I sent you in a private message some days ago:

@Sebastian-Roth said:

Seems like the REMI GPG key changed. You need to manually run dnf update once and confirm the GPG key import.
Interesting this is not happening for the dev-branch tests.

I still try to get things ready for a very last 1.5.x release. As my time is kind of limited to work on this it’s just taking ages. Still can’t give you a definite date on when it’s all ready. Anyone is welcome to contact me and help getting the last bits done. Although I have to say that good knowledge of FOG is needed to be of help.

PS: As CentOS is fading we need to look at the successors as well. I did a test install on Rocky 8 as well as AlmaLinux 8 and found both worked. So we are on a good road with those as well.

Wayne Workman

@sebastian-roth said in RHEL 8, CentOS 8, master branch, July 22, 2021:

Seems like the REMI GPG key changed. You need to manually run dnf update once and confirm the GPG key import.
Interesting this is not happening for the dev-branch tests.

@Sebastian-Roth I have done this. Which is why I’m bringing it up again.

Sebastian Roth

@Wayne-Workman I am pretty sure I did a plain test install with 1.5.9 on CentOS 8 and it worked. Will try again.

Sebastian Roth

@Wayne-Workman Figured that you can’t confirm the GPG key for REMI’s repo when running the installer in silent mode (confirm all questions). We do import the key via rpm --import ... via the installer script but turns out this is not enough when dnf is used because it uses a different gpg database “by design”: https://bugzilla.redhat.com/show_bug.cgi?id=1768206#c4

Do you have some kind of mechanism in your test setup that allows to run some commands before the installer test? You might run this command on CentOS 8 and RHEL 8 for the time being.

sed -i -e 's/dnf check-update/dnf -y check-update/g' /path/to/fogproject/lib/redhat/config.sh

As I said, we work on the next release but can’t give you a date yet.

Looking at the test results today we see that dev-branch fails on CentOS 8 and RHEL 8 as well. Strange error for RHEL 8:

package remi-release-8* is not installed
warning: /var/tmp/rpm-tmp.da0Ljt: Header V4 RSA/SHA256 Signature, key ID 5f11735a: NOKEY
Retrieving https://rpms.remirepo.net/enterprise/remi-release-8.rpm
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
remi-release-8.4-1.el8.remi           ########################################
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to rpms.remirepo.net:443 
error: https://rpms.remirepo.net/RPM-GPG-KEY-remi: import read failed(2).

And similar for CentOS 8:

package remi-release-8* is not installed
curl: (52) Empty reply from server
error: skipping https://rpms.remirepo.net/enterprise/remi-release-8.rpm - transfer failed
Retrieving https://rpms.remirepo.net/enterprise/remi-release-8.rpm

And even CentOS 7:

package remi-release-7* is not installed
curl: (56) TCP connection reset by peer
error: skipping https://rpms.remirepo.net/enterprise/remi-release-7.rpm - transfer failed
Retrieving https://rpms.remirepo.net/enterprise/remi-release-7.rpm

Probably a network issue…

Wayne Workman

@sebastian-roth Yeah, today’s tests look like a mess. Though, what your saying about this GPG key. All the tests use the same OS, same patch level each day. If there was an issue with a GPG key in master branch, wouldn’t it be true that the same problem exists in dev branch? Most days, this is not the case. Most days, just master branch fail for CentOS 8 and RHEL 8.

There are ways to inject commands before installation, though the point of these tests is just to see in an automatic way if there are any issues that arise for a typical installation.

Wayne Workman

Today pictured below, for example. This is what normally happens. Master fails and dev-branch succeeds. Yet, dev-branch and master branch are executing against the same OS, same patch level. The way these tests are setup is - the VMs get a snapshot restored. Then they are patched. Then they get the specified branch installed. This is repeated for each branch. Fresh snapshot applied and fresh patching performed before every installation.

CentOS 8 Output for master branch: 2021-07-30_12-00-PM_output.log

CentOS 8 Error Log for master branch: 2021-07-30_12-00-PM_fog_error.log

dev-branch fog_error output for CentOS 8:

package remi-release-8* is not installed
warning: /var/tmp/rpm-tmp.AVYE4j: Header V4 RSA/SHA256 Signature, key ID 5f11735a: NOKEY
Retrieving https://rpms.remirepo.net/enterprise/remi-release-8.rpm
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
remi-release-8.4-1.el8.remi           ########################################
Extra Packages for Enterprise Linux Modular 8 - 1.9 MB/s | 798 kB     00:00    
Extra Packages for Enterprise Linux 8 - x86_64   24 MB/s |  10 MB     00:00    
Remi's Modular repository for Enterprise Linux  772  B/s | 858  B     00:01    
Remi's Modular repository for Enterprise Linux  3.0 MB/s | 3.1 kB     00:00    
Importing GPG key 0x5F11735A:
 Userid     : "Remi's RPM repository <remi@remirepo.net>"
 Fingerprint: 6B38 FEA7 231F 87F5 2B9C A9D8 5550 9759 5F11 735A
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el8
Remi's Modular repository for Enterprise Linux  795 kB/s | 779 kB     00:00    
Safe Remi's RPM repository for Enterprise Linux 1.6 kB/s | 858  B     00:00    
Safe Remi's RPM repository for Enterprise Linux 3.0 MB/s | 3.1 kB     00:00    
Importing GPG key 0x5F11735A:
 Userid     : "Remi's RPM repository <remi@remirepo.net>"
 Fingerprint: 6B38 FEA7 231F 87F5 2B9C A9D8 5550 9759 5F11 735A
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el8
Safe Remi's RPM repository for Enterprise Linux 1.9 MB/s | 1.9 MB     00:01    
bc-1.07.1-5.el8.x86_64
curl-7.61.1-18.el8.x86_64

master branch fog_error output for CentOS8:

package remi-release-8* is not installed
warning: /var/tmp/rpm-tmp.fFeNbj: Header V4 RSA/SHA256 Signature, key ID 5f11735a: NOKEY
Retrieving http://rpms.remirepo.net/enterprise/remi-release-8.rpm
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
remi-release-8.4-1.el8.remi           ########################################
Extra Packages for Enterprise Linux Modular 8 - 4.7 MB/s | 798 kB     00:00    
Extra Packages for Enterprise Linux 8 - x86_64   32 MB/s |  10 MB     00:00    
Remi's Modular repository for Enterprise Linux  568  B/s | 858  B     00:01    
Remi's Modular repository for Enterprise Linux  3.0 MB/s | 3.1 kB     00:00    
Importing GPG key 0x5F11735A:
 Userid     : "Remi's RPM repository <remi@remirepo.net>"
 Fingerprint: 6B38 FEA7 231F 87F5 2B9C A9D8 5550 9759 5F11 735A
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el8
Is this ok [y/N]: Remi's Modular repository for Enterprise Linux  2.4 kB/s | 858  B     00:00    
Error: Failed to download metadata for repo 'remi-modular': repomd.xml GPG signature verification error: Bad GPG signature
bc-1.07.1-5.el8.x86_64
curl-7.61.1-18.el8.x86_64

Firefox_Screenshot_2021-07-30T17-39-55.385Z.png

Wayne Workman

The difference between dev-branch and master branch appears to be http vs https.

master:
https://github.com/FOGProject/fogproject/blob/171d63724131c396029992730660497d48410842/lib/common/functions.sh#L658

dev-branch:
https://github.com/FOGProject/fogproject/blob/3a3ad6cda7cf379f0c3493ee39a4e27a08e24e69/lib/common/functions.sh#L665

Here’s the commit that made the change:
https://github.com/FOGProject/fogproject/commit/349a8fa2df00d82df857527b1cf45aa204363834

Introduced in 1.5.9.58 of dev-branch

Sebastian Roth

@sebastian-roth said in RHEL 8, CentOS 8, master branch, July 22, 2021:

sed -i -e 's/dnf check-update/dnf -y check-update/g' /path/to/fogproject/lib/redhat/config.

Not sure if you saw that I edited my initial post. That sed command should take care of the issue that is causing master to fail. Find the related commit that fixed the issue in dev-branch here: https://github.com/FOGProject/fogproject/commit/90caf2367d21d0594b7f16b2813a2fa94684ad78

RHEL 8, CentOS 8, master branch, July 22, 2021

256

11.6k

17.1k

154.6k