• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

rcu_sched stall OR kernel panic on PowerEdge R640

Scheduled Pinned Locked Moved Solved
FOG Problems
5
45
5.0k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    Sebastian Roth Moderator
    last edited by Sep 26, 2019, 12:45 PM

    @george1421 said in rcu_sched stall OR kernel panic on PowerEdge R640:

    We can also hold this “test” kernel in reserve in case this issue comes up again if you don’t want to release it as general availability.

    Don’t get me wrong on this. I am more than happy to make this the default kernel for everyone. It comes at low cost. But I’d like to see this tested on several different machines (PC as well as notebooks and even servers if possible) before we make it the new default kernel.

    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

    1 Reply Last reply Reply Quote 2
    • Q
      Quazz Moderator @Sebastian Roth
      last edited by Sep 26, 2019, 1:25 PM

      @Sebastian-Roth As far as I understand it, Xenomai implements a patch to the kernel that does all kinds of stuff, potentially it’s not compatible with their patches, but as far as I know CONFIG_MAXSMP is in fact enabled by default on Kernel 4.4+ or so on all major distributions without issues.

      That said, I don’t mind testing it.

      1 Reply Last reply Reply Quote 0
      • S
        Sebastian Roth Moderator
        last edited by Sep 26, 2019, 1:28 PM

        @Quazz said in rcu_sched stall OR kernel panic on PowerEdge R640:

        as far as I know CONFIG_MAXSMP is in fact enabled by default on Kernel 4.4+ or so on all major distributions without issues

        That’s valuable information! Any reference for this?

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        Q 1 Reply Last reply Oct 2, 2019, 12:37 PM Reply Quote 0
        • Q
          Quazz Moderator @Sebastian Roth
          last edited by Quazz Oct 2, 2019, 6:38 AM Oct 2, 2019, 12:37 PM

          @Sebastian-Roth Hmm, I may have been misremembering, though their CONFIG_NR_CPUS
          is going to be much higher than 8 at the very least. (at least 512 afaik)

          The only difference I can find is that CONFIG_MAXSMP enables CPUMASK_OFFSTACK
          , which it requires to function correctly I believe (or any high CONFIG_NR_CPUS would at least)

          1 Reply Last reply Reply Quote 0
          • S
            Sebastian Roth Moderator
            last edited by Oct 21, 2019, 8:46 PM

            @Quazz @george1421 Ok, back from travels… what shall we do with this pending topic. I do understand that adding CONFIG_MAXSMP does fix the rcu_sched stall issue on PowerEdge R640. But do we know if this fixes rcu_sched stalls on other platforms as well? Would we get at least two more people to test this before we add it to the official kernel?

            @george1421 Did you get to test this kernel on your fleet of Dell hardware to see if it might cause any other harm?

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            G 1 Reply Last reply Oct 21, 2019, 8:50 PM Reply Quote 0
            • G
              george1421 Moderator @Sebastian Roth
              last edited by Oct 21, 2019, 8:50 PM

              @Sebastian-Roth said in rcu_sched stall OR kernel panic on PowerEdge R640:

              Did you get to test this kernel on your fleet of Dell hardware to see if it might cause any other harm?

              TBH, no I did not test it. I haven’t found any other system that the max-cpu value fixed either. We had one dual core with the rcu_sched stall, but that was fixed with the current kernel and changing the acpi clock source that Quazz posted. I think the max-cpu will only impact CPUs with more than 8 cores.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

              1 Reply Last reply Reply Quote 0
              • S
                Sebastian Roth Moderator
                last edited by Oct 21, 2019, 9:05 PM

                @george1421 Looking through a stack of other rcu_sched stall topics in the forums I can’t seem to find any thread where I’d think that people had CPUs with more than 8 cores. Sure sooner or later this will be state of the art but I don’t reckon we should step ahead of this. We know the current kernel works pretty good on most CPUs and I’d rather point people to this topic and provide compile instructions than setting CONFIG_MAXSMP as default. Hmm?

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                G 1 Reply Last reply Oct 21, 2019, 9:30 PM Reply Quote 0
                • G
                  george1421 Moderator @Sebastian Roth
                  last edited by Oct 21, 2019, 9:30 PM

                  @Sebastian-Roth I’m still on the fence about this, I would say turn it on because the core count continues to rise on these processors. What can it hurt? And on the other side we really don’t know what the impact could be.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                  1 Reply Last reply Reply Quote 0
                  • S
                    Sebastian Roth Moderator
                    last edited by Oct 21, 2019, 9:42 PM

                    @george1421 said in rcu_sched stall OR kernel panic on PowerEdge R640:

                    And on the other side we really don’t know what the impact could be.

                    Yes, because of that I don’t like switching it on.

                    What can it hurt?

                    I don’t know. Stalls on older CPUs?

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    Q 1 Reply Last reply Oct 22, 2019, 7:47 AM Reply Quote 0
                    • Q
                      Quazz Moderator @Sebastian Roth
                      last edited by Quazz Oct 22, 2019, 1:54 AM Oct 22, 2019, 7:47 AM

                      @Sebastian-Roth

                      There are kernel flags to disable SMP if necessary, so I think it’s pretty safe to compile with MAXSMP. Just my opinion of course; without a diverse test fleet it’s hard to say for sure since kernels can always have bugs or unforeseen interactions. But that would be true for any change we make.

                      I can’t find anything googling about stalls/problems with MAXSMP either. Only some people on embedded systems who want to reduce the size of their kernel, but that’s a targetted compile anyway.

                      There will be more and more systems entering the floor with more than 8 cores (our current NR_CPU value) given the recent CPU releases as well, so at the very least that number could use a bump.

                      1 Reply Last reply Reply Quote 0
                      • S
                        Sebastian Roth Moderator
                        last edited by Oct 22, 2019, 10:16 AM

                        @Quazz said in rcu_sched stall OR kernel panic on PowerEdge R640:

                        I can’t find anything googling about stalls/problems with MAXSMP either. Only some people on embedded systems who want to reduce the size of their kernel, but that’s a targetted compile anyway.

                        Ok, you and George have convinced me this is most probably not going to cause us much trouble, so I will add the options as mentioned below.

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        1 Reply Last reply Reply Quote 0
                        • S
                          Sebastian Roth Moderator
                          last edited by Oct 22, 2019, 9:03 PM

                          @george1421 @Quazz I just added the two kernel options as mentioned below to the x64 kernel config (not pushed the change yet).

                          While CONFIG_INTEL_IDLE is available in x86 (32 bit) config as well CONFIG_MAXSMP is not (depends on X86_64 [=y]). Should I leave CONFIG_NR_CPUS set to 8 (default I think) or increase it to 16, 32, 64?

                          For ARM kernel config we don’t have CONFIG_INTEL_IDLE nor CONFIG_MAXSMP but can adjust CONFIG_NR_CPUS too. 16, 32, 64?

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          Q 1 Reply Last reply Oct 23, 2019, 7:46 AM Reply Quote 0
                          • Q
                            Quazz Moderator @Sebastian Roth
                            last edited by Quazz Oct 23, 2019, 1:48 AM Oct 23, 2019, 7:46 AM

                            @Sebastian-Roth I think it’s fine to leave X86 to 8 since I don’t think they make huge multicore 32 bit CPUs. (wouldn’t really make sense to me anyway) Though I also think it wouldn’t necessarily hurt to change it, why bother if it’s not needed? I also believe 8 is default for X86 anyway

                            As for ARM: https://www.phoronix.com/scan.php?page=news_item&px=ARM64-256-Default-NR_CPUS

                            Default in Linux 5.1 ARM 64 is 256 now. (current default being 64)

                            1 Reply Last reply Reply Quote 1
                            • S
                              Sebastian Roth Moderator
                              last edited by Oct 23, 2019, 11:30 AM

                              @Quazz Thanks, done: https://github.com/FOGProject/fos/issues/31

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              1 Reply Last reply Reply Quote 1
                              • 1
                              • 2
                              • 3
                              • 3 / 3
                              • First post
                                Last post

                              144

                              Online

                              12.0k

                              Users

                              17.3k

                              Topics

                              155.2k

                              Posts
                              Copyright © 2012-2024 FOG Project