• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

RHEL 8, CentOS 8, master branch, July 22, 2021

Scheduled Pinned Locked Moved
General
2
9
1.8k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • W
    Wayne Workman
    last edited by Wayne Workman Jul 22, 2021, 10:04 PM Jul 23, 2021, 3:54 AM

    I’ve been monitoring & have once cleaned up the daily installation tests, and for a while now RHEL 8 and CentOS 8 have been failing every day on the master branch, while the dev branch passes tests every day.

    Given the dev branch is working fine, it would seem the fix is to release the current dev as a new version. What sort of shape is the dev branch concerning all the other aspects of FOG? Is it in a shape to release?

    The external reporting graphs show 50 servers on 1.5.9.98, with overall about 460 servers operating on the dev branch.

    There is another aspect to this - maybe we don’t prioritize a release just for fixing CentOS 8 and RHEL8. It’s a legitimate question to ask, at least. External reporting shows CentOS 8 and CentOS Stream 8 being around 15 total for the dev branch. Pretty low figure. RHEL doesn’t even register in the top 20 OS versions in use. Raspbian has a higher adoption on dev branch than RHEL.

    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
    Daily Clean Installation Results:
    https://fogtesting.fogproject.us/
    FOG Reporting:
    https://fog-external-reporting-results.fogproject.us/

    1 Reply Last reply Reply Quote 0
    • S
      Sebastian Roth Moderator
      last edited by Sebastian Roth Jul 25, 2021, 3:55 PM Jul 25, 2021, 9:54 PM

      @Wayne-Workman Thanks for bringing this up. Before I answer your question on the next release I might quote what I sent you in a private message some days ago:

      @Sebastian-Roth said:

      Seems like the REMI GPG key changed. You need to manually run dnf update once and confirm the GPG key import.
      Interesting this is not happening for the dev-branch tests.

      I still try to get things ready for a very last 1.5.x release. As my time is kind of limited to work on this it’s just taking ages. Still can’t give you a definite date on when it’s all ready. Anyone is welcome to contact me and help getting the last bits done. Although I have to say that good knowledge of FOG is needed to be of help.

      PS: As CentOS is fading we need to look at the successors as well. I did a test install on Rocky 8 as well as AlmaLinux 8 and found both worked. So we are on a good road with those as well.

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      W 1 Reply Last reply Jul 28, 2021, 5:25 PM Reply Quote 1
      • W
        Wayne Workman @Sebastian Roth
        last edited by Jul 28, 2021, 5:25 PM

        @sebastian-roth said in RHEL 8, CentOS 8, master branch, July 22, 2021:

        Seems like the REMI GPG key changed. You need to manually run dnf update once and confirm the GPG key import.
        Interesting this is not happening for the dev-branch tests.

        @Sebastian-Roth I have done this. Which is why I’m bringing it up again.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
        Daily Clean Installation Results:
        https://fogtesting.fogproject.us/
        FOG Reporting:
        https://fog-external-reporting-results.fogproject.us/

        1 Reply Last reply Reply Quote 0
        • S
          Sebastian Roth Moderator
          last edited by Jul 28, 2021, 6:28 PM

          @Wayne-Workman I am pretty sure I did a plain test install with 1.5.9 on CentOS 8 and it worked. Will try again.

          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

          1 Reply Last reply Reply Quote 0
          • S
            Sebastian Roth Moderator
            last edited by Sebastian Roth Jul 29, 2021, 4:04 PM Jul 29, 2021, 8:56 PM

            @Wayne-Workman Figured that you can’t confirm the GPG key for REMI’s repo when running the installer in silent mode (confirm all questions). We do import the key via rpm --import ... via the installer script but turns out this is not enough when dnf is used because it uses a different gpg database “by design”: https://bugzilla.redhat.com/show_bug.cgi?id=1768206#c4

            Do you have some kind of mechanism in your test setup that allows to run some commands before the installer test? You might run this command on CentOS 8 and RHEL 8 for the time being.

            sed -i -e 's/dnf check-update/dnf -y check-update/g' /path/to/fogproject/lib/redhat/config.sh
            

            As I said, we work on the next release but can’t give you a date yet.

            Looking at the test results today we see that dev-branch fails on CentOS 8 and RHEL 8 as well. Strange error for RHEL 8:

            package remi-release-8* is not installed
            warning: /var/tmp/rpm-tmp.da0Ljt: Header V4 RSA/SHA256 Signature, key ID 5f11735a: NOKEY
            Retrieving https://rpms.remirepo.net/enterprise/remi-release-8.rpm
            Verifying...                          ########################################
            Preparing...                          ########################################
            Updating / installing...
            remi-release-8.4-1.el8.remi           ########################################
            curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to rpms.remirepo.net:443 
            error: https://rpms.remirepo.net/RPM-GPG-KEY-remi: import read failed(2).
            

            And similar for CentOS 8:

            package remi-release-8* is not installed
            curl: (52) Empty reply from server
            error: skipping https://rpms.remirepo.net/enterprise/remi-release-8.rpm - transfer failed
            Retrieving https://rpms.remirepo.net/enterprise/remi-release-8.rpm
            

            And even CentOS 7:

            package remi-release-7* is not installed
            curl: (56) TCP connection reset by peer
            error: skipping https://rpms.remirepo.net/enterprise/remi-release-7.rpm - transfer failed
            Retrieving https://rpms.remirepo.net/enterprise/remi-release-7.rpm
            

            Probably a network issue…

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            W 1 Reply Last reply Jul 29, 2021, 11:13 PM Reply Quote 0
            • W
              Wayne Workman @Sebastian Roth
              last edited by Wayne Workman Jul 29, 2021, 5:14 PM Jul 29, 2021, 11:13 PM

              @sebastian-roth Yeah, today’s tests look like a mess. Though, what your saying about this GPG key. All the tests use the same OS, same patch level each day. If there was an issue with a GPG key in master branch, wouldn’t it be true that the same problem exists in dev branch? Most days, this is not the case. Most days, just master branch fail for CentOS 8 and RHEL 8.

              There are ways to inject commands before installation, though the point of these tests is just to see in an automatic way if there are any issues that arise for a typical installation.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
              Daily Clean Installation Results:
              https://fogtesting.fogproject.us/
              FOG Reporting:
              https://fog-external-reporting-results.fogproject.us/

              1 Reply Last reply Reply Quote 0
              • W
                Wayne Workman
                last edited by Wayne Workman Jul 30, 2021, 11:50 AM Jul 30, 2021, 5:42 PM

                Today pictured below, for example. This is what normally happens. Master fails and dev-branch succeeds. Yet, dev-branch and master branch are executing against the same OS, same patch level. The way these tests are setup is - the VMs get a snapshot restored. Then they are patched. Then they get the specified branch installed. This is repeated for each branch. Fresh snapshot applied and fresh patching performed before every installation.

                CentOS 8 Output for master branch: 2021-07-30_12-00-PM_output.log

                CentOS 8 Error Log for master branch: 2021-07-30_12-00-PM_fog_error.log

                dev-branch fog_error output for CentOS 8:

                package remi-release-8* is not installed
                warning: /var/tmp/rpm-tmp.AVYE4j: Header V4 RSA/SHA256 Signature, key ID 5f11735a: NOKEY
                Retrieving https://rpms.remirepo.net/enterprise/remi-release-8.rpm
                Verifying...                          ########################################
                Preparing...                          ########################################
                Updating / installing...
                remi-release-8.4-1.el8.remi           ########################################
                Extra Packages for Enterprise Linux Modular 8 - 1.9 MB/s | 798 kB     00:00    
                Extra Packages for Enterprise Linux 8 - x86_64   24 MB/s |  10 MB     00:00    
                Remi's Modular repository for Enterprise Linux  772  B/s | 858  B     00:01    
                Remi's Modular repository for Enterprise Linux  3.0 MB/s | 3.1 kB     00:00    
                Importing GPG key 0x5F11735A:
                 Userid     : "Remi's RPM repository <remi@remirepo.net>"
                 Fingerprint: 6B38 FEA7 231F 87F5 2B9C A9D8 5550 9759 5F11 735A
                 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el8
                Remi's Modular repository for Enterprise Linux  795 kB/s | 779 kB     00:00    
                Safe Remi's RPM repository for Enterprise Linux 1.6 kB/s | 858  B     00:00    
                Safe Remi's RPM repository for Enterprise Linux 3.0 MB/s | 3.1 kB     00:00    
                Importing GPG key 0x5F11735A:
                 Userid     : "Remi's RPM repository <remi@remirepo.net>"
                 Fingerprint: 6B38 FEA7 231F 87F5 2B9C A9D8 5550 9759 5F11 735A
                 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el8
                Safe Remi's RPM repository for Enterprise Linux 1.9 MB/s | 1.9 MB     00:01    
                bc-1.07.1-5.el8.x86_64
                curl-7.61.1-18.el8.x86_64
                

                master branch fog_error output for CentOS8:

                package remi-release-8* is not installed
                warning: /var/tmp/rpm-tmp.fFeNbj: Header V4 RSA/SHA256 Signature, key ID 5f11735a: NOKEY
                Retrieving http://rpms.remirepo.net/enterprise/remi-release-8.rpm
                Verifying...                          ########################################
                Preparing...                          ########################################
                Updating / installing...
                remi-release-8.4-1.el8.remi           ########################################
                Extra Packages for Enterprise Linux Modular 8 - 4.7 MB/s | 798 kB     00:00    
                Extra Packages for Enterprise Linux 8 - x86_64   32 MB/s |  10 MB     00:00    
                Remi's Modular repository for Enterprise Linux  568  B/s | 858  B     00:01    
                Remi's Modular repository for Enterprise Linux  3.0 MB/s | 3.1 kB     00:00    
                Importing GPG key 0x5F11735A:
                 Userid     : "Remi's RPM repository <remi@remirepo.net>"
                 Fingerprint: 6B38 FEA7 231F 87F5 2B9C A9D8 5550 9759 5F11 735A
                 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-remi.el8
                Is this ok [y/N]: Remi's Modular repository for Enterprise Linux  2.4 kB/s | 858  B     00:00    
                Error: Failed to download metadata for repo 'remi-modular': repomd.xml GPG signature verification error: Bad GPG signature
                bc-1.07.1-5.el8.x86_64
                curl-7.61.1-18.el8.x86_64
                

                Firefox_Screenshot_2021-07-30T17-39-55.385Z.png

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                Daily Clean Installation Results:
                https://fogtesting.fogproject.us/
                FOG Reporting:
                https://fog-external-reporting-results.fogproject.us/

                1 Reply Last reply Reply Quote 0
                • W
                  Wayne Workman
                  last edited by Wayne Workman Jul 30, 2021, 11:58 AM Jul 30, 2021, 5:54 PM

                  The difference between dev-branch and master branch appears to be http vs https.

                  master:
                  https://github.com/FOGProject/fogproject/blob/171d63724131c396029992730660497d48410842/lib/common/functions.sh#L658

                  dev-branch:
                  https://github.com/FOGProject/fogproject/blob/3a3ad6cda7cf379f0c3493ee39a4e27a08e24e69/lib/common/functions.sh#L665

                  Here’s the commit that made the change:
                  https://github.com/FOGProject/fogproject/commit/349a8fa2df00d82df857527b1cf45aa204363834

                  Introduced in 1.5.9.58 of dev-branch

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                  Daily Clean Installation Results:
                  https://fogtesting.fogproject.us/
                  FOG Reporting:
                  https://fog-external-reporting-results.fogproject.us/

                  1 Reply Last reply Reply Quote 0
                  • S
                    Sebastian Roth Moderator
                    last edited by Sebastian Roth Jul 31, 2021, 4:20 AM Jul 31, 2021, 10:20 AM

                    @sebastian-roth said in RHEL 8, CentOS 8, master branch, July 22, 2021:

                    sed -i -e 's/dnf check-update/dnf -y check-update/g' /path/to/fogproject/lib/redhat/config.
                    

                    Not sure if you saw that I edited my initial post. That sed command should take care of the issue that is causing master to fail. Find the related commit that fixed the issue in dev-branch here: https://github.com/FOGProject/fogproject/commit/90caf2367d21d0594b7f16b2813a2fa94684ad78

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    1 Reply Last reply Reply Quote 1
                    • 1 / 1
                    1 / 1
                    • First post
                      1/9
                      Last post

                    214

                    Online

                    12.1k

                    Users

                    17.3k

                    Topics

                    155.3k

                    Posts
                    Copyright © 2012-2024 FOG Project