FOG - boot to FOS - rcu_sched self-detected stall on CPU

[abos_systemax]

Server

• FOG Version: 3 RC21
• OS: Debian Jessy

Client

• Lenovo E560
• BIOS: Legacy

Description

Hello,

When I try to boot to the Quick Host Registration on FOG, I see a screen with the following info:

INFO: rcu_sched self-detected stall on CPU
o0-…: (337893 ticks this GP) idle=439/14000000000001/0 softirq=119/146 fqs=83192
o (t=336015 jiffies g=-274 c=-275 q=168)

This info displays every so often with different numbers every time.

bzImage Version: 4.8.1
bzImage32 Version: 4.8.1

it boots using the bzImage32

edit: just updated to RC21, problem still exists
edit2: changed the title, the issue is not only for the Quick reg, it’s the entire FOG OS that doesn’t boot

[Tom Elliott]

I don’t understand. Is it preventing functionality? Or just a message you’re asking about?

[abos_systemax]

It’s the fact that it doesn’t do anything but displaying the message over and over again, so it does prevent functionality.

I have found references online to a BIOS setting named AMD C1E, but this is an Intel board, so that might not be the solution; maybe there is an equivalent to it…

However, this message should not stall the device I’d assume based on your reply?

[Tom Elliott]

@abos_systemax I don’t, I’m just asking a question.

Maybe try enabling/disabling VTx? I’m grasping at straws though.

I suspect, this also happens under full registration?

[abos_systemax]

@Tom-Elliott
It does indeed stall in the Full registration as well.
Enableling VT/VTd doesnt resolve the issue

[Tom Elliott]

Do you have another of the same type of system? Just wanting to know if this is a one off case, or if all systems of the same type have the same problem.

[Tom Elliott]

I ask this because:
https://www.kernel.org/doc/Documentation/RCU/stallwarn.txt

[abos_systemax]

@Tom-Elliott
I was trying to understand that document when I found it on google, but am unfamilair with all the abbreviations used, I am also a ‘transitioner’ from Windows to Linux, so it isn’t my forté yet.

I have more devices of the kind, will test as soon as I receive the other from our warehouse

[abos_systemax]

@Tom-Elliott it does appear to be model related because the other device also isn’t able to boot (with the same rcu_sched messages)

[Tom Elliott]

Can you try changing the kernel out for one of the 4.6 kernels? 4.7 I added a bunch more “stuff” which may have included RCU_Scheduling. 4.6, I’m pretty certain, did not have the “extras” and may help you out here.

[abos_systemax]

@Tom-Elliott both 4.6.4 and 4.5.0 give the same error as the 4.8.1 bzImage32s

e1: i even went as far as downgrading to 3.0.1, but that one results in a blinking cursor on the top left
e2: as does 4.1.0

e3: 4.2.0 is the first kernel to display the rcu_sched error apparently

[abos_systemax]

@Tom-Elliott
ok, so as soon as I disable Hyper-treading (which makes ipxe monstrously slow btw), I still receive the Informational error, but immediatly after that I receive an rcu_sched kthread starved for xxxxxx jiffies! (where xxxx is a number)
_RCU_GP_WAIT_FQS(3) -> state=0x1 (and the following message is state=0x0)

[Tom Elliott]

@abos_systemax If you re-up to the current kernel and set the log level down (from FOG Configuration->FOG Settings->FOG Boot Settings->FOG_KERNEL_LOGLEVEL) I imagine you will see less of these messages?

[abos_systemax]

@Tom-Elliott setting it to 0 gives me the blinking cursor, 1 as well
bumping it up to Debug and log level 7, logs until PCI device initialisation before the RCU informationals appear and I then receive a task dump for CPU 0

bumping it back down to level 4 doesn’t show less messages, so apparently Log level 7 is the same as log level 4 with Kernel_debug?

btw there is a small typo in the helptext for loglevel (the instead of they)

[abos_systemax]

@Tom-Elliott There isn’t a firmware update available for these machines either. It’s on 1.22 and Lenovo’s latest release is 1.22

[abos_systemax]

I can also confirm that other brands of Linux are able too boot.

[abos_systemax]

@Tom-Elliott

as a matter of debugging, I tried booting to UEFI…
Then I receive the iPXE error 0x7f048283, which is - funnily enough - the same error I had yesterday on a Lenovo M700 which wás able to boot on Legacy

[Tom Elliott]

@abos_systemax Loglevel is independent of the “Kernel Debug”

Kernel Debug will turn on all debug messages. Loglevel will automatically be set to full for the kernel debug regardless of what you set the loglevel to.

I doubt it’s firmware related. The messages, in and of themselves is fine.

Is this booting bzImage or bzImage32?

Is the system in UEFI or Legacy?

[abos_systemax]

@Tom-Elliott the device displays the INFOrmational; when in Legacy (and booting bzImage32), but displays the 0x7f048283 when in UEFI
If i force the bzImage on legacy then … =it works=

[Tom Elliott]

@abos_systemax Erm, why is it booting bzImage32 for legacy but bzImage for uefi?

32 is only for 32 bit requests.