Getting a boot error. Kernel panic for block (0,0)

[gribbler]

It seems like fog init.xz is unable to load the drivers for the local drive on the system. I’ve downloaded newer kernel and init.xz and it still persists, and I feel like I’m missing something obvious. Thanks for taking a look

CPU - AMD Ryzen Threadripper 3970X 32-Core
1tb samsung 870 ssd.

fstab on host
/dev/mapper/rl-root / xfs defaults 0 0
UUID=811b72e5-eba6-446a-b8f7-90cbc88c6d08 /boot xfs defaults 0 0
UUID=E788-250F /boot/efi vfat umask=0077,shortname=winnt 0 2
/dev/mapper/rl-home /home xfs defaults 0 0
/dev/mapper/rl-swap none swap defaults 0 0

I am getting the following error, edited slightly

0800 drive-size sda
driver: sd

0801 partition size sda1 UUID?

0802 partition size sda2 UUID?

0803 partition size sda3 UUID?

List of all bdev filesytems
ext3
ext4
ext2
vfat
msdos
exfat
hfsplus
hfs
ntfs
fuseblk
xfs
btrfs

kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
Kernel Offset: disabled

[gribbler]

Is this possibly an issue with AHCI not being built into the kernel? That seems unlikely but is my latest hole i’m running down

[george1421]

@gribbler Can we get an actual screen shot of the error? The context of the error is almost is as important as the actual error itself.

When you get the kernel panic is it when FOS linux (bzImage) is booting or your target OS? Typically I’ve seen this when FOS linux engine is booting and for some reason bzImage doesn’t match init.xz file format. Such as if you use a new init.xz file with an old FOS linux kernel or the init.xz file isn’t created by the fog project developers.

[gribbler]

@george1421

Thanks George,

Rocky 9 is the OS.
This was with the default bzImage and init.xz that the system installed, as well I downloaded a couple a version of each from the kernel and initrd sections in the Fog Configuration. I could be just missing something obvious, I just found it odd that the 1 system I’ve chosen to use as my image source, turns out to be unbootable with fog. I’ve had very limited experience with Fog in the past, and it just worked - that was at home on my 4 pc home lab.

I was unable to find a .config file from the fog kernels so I wasn’t sure if AHCI was loading as a module or not and wondering if that was a problem… You don’t want me mashing about trying to build a kernel.

Wrong screen cap attached - editing message:

[george1421]

@gribbler While I understand where its failing in the linux kernel booting process, I’ve never see the FOS Linux kernel do that. Just for clarity is this the target OS booting creating this message or is it happening when you begin to capture/deploy an image?

[gribbler]

This is as the fos kernel boots, shortly after the bzImage and init.xz have downloaded and the boot process had started. Before it, there’s a screen or two of the boot process though it’s a bit too quick to give details but I’m not seeing anything that stands out in the bits I get to read

[george1421]

@gribbler Something is telling me the bzImage and init.xz files are out of sync or init.xz is damage in some way.

Looking at your OP and the screen cap I still can’t believe that this is FOS linux starting up. Especially this part:

fstab on host
/dev/mapper/rl-root / xfs defaults 0 0
UUID=811b72e5-eba6-446a-b8f7-90cbc88c6d08 /boot xfs defaults 0 0
UUID=E788-250F /boot/efi vfat umask=0077,shortname=winnt 0 2
/dev/mapper/rl-home /home xfs defaults 0 0
/dev/mapper/rl-swap none swap defaults 0 0

The xfs file system and the second partition having a short name of winnt that’s not FOS Linux. The fos linux filesystem in init.xz is ext2 or ext3, I can’t remember, but its surely not xfs. This makes me think its the target computer’s OS for some reason.

Working with the understanding that this is truly a FOS linux boot, lets have you reinstall FOG. Simply rerun the installfog.sh installer. It will take all of your previous settings and won’t touch the database, but it will reset all of the files in the fog server. Once that is done then use the web ui fog configuration->kernel update to bring the kernel up to 6.6.x. Don’t do anything by hand in the fog server host computer’s file system. This should restore the files back to a known state.

I didn’t happen to ask you what version of FOG you are using, hopefully is 5.10.x

Lastly about your ahci question, yes that driver has always been in the fog kernel. But if you want to look the kernel configs are located here: https://github.com/FOGProject/fos/tree/master/configs

[gribbler]

Thank you, that’s helpful for my understanding of a few things. My guess was that the FOS was booting and then trying to load the system ssd drive, and crapping out there. I will get a further look at it tomorrow at the office. It’s odd that shortname is there, this was a new ssd pulled out about 3 weeks ago as far as i know and for sure only has had rocky9 workstation installed on it. I had rebuilt the fog server from scratch earlier this week on a new vm just to make sure I was starting from fresh after screwing around with building my own init.xz and bzImage - after struggling for a while with what I thought was a different IPXE problem, when it turns out it was a bug in the switch firmware that was turning the network port off on some reboots unless it was set to 1g b(from 10gb).

I can recreate the server, and reinstall the OS on the workstation on the driver after i delete all the partitions just to get things as simple as possible. Thanks again.

[gribbler]

@george1421 Thanks again for your reply.

I rebuilt the server, though I need to use an updated ipxe.efi to support the network card (Marvell AQC107).

This gets me to the exact same place I was with the kernel panic.

Just to be clear - is my assumption correct? it seems like in the ipxe boot process, the fog kernel is crashing when trying to mount the local host OS drive. I have to step away for a few hours and perhaps not get back to it until tomorrow.

thanks.

[gribbler]

@gribbler I’ve rebuilt the workstation a couple times, once with defaults for the system drive and the other with manually settings ext3 partitioning, and still getting the same failure.

I’m stumped as to why this system is not booting. I’ll see if I can steal a system from someone else and try theirs, it might be something odd about the hardware.

[gribbler]

Bump for some love…

[Tom Elliott]

@gribbler Based on what I can only guess is a question unrelated to FOS itself:

FOG doesn’t do anything related to fstab on any host.

So this tells me what you’re seeing is completely seperate and most likely the drive booting into normal operating system of which is using xfs filesystm.

Please make a backup of your init.xz/bzImage files and download these specific files:

cp /var/www/fog/service/ipxe/bzImage{,_backup}
cp /var/www/fog/service/ipxe/init.xz{,_backup}
wget -O /var/www/fog/service/ipxe/bzImage https://github.com/fogproject/fos/releases/latest/download/bzImage
wget -O /var/www/fog/service/ipxe/init.xz https://github.com/fogproject/fos/releases/latest/download/init.xz

Then try loading fog?

If you can also capture a video of the whole process and post it somewhere we can try to get to it?

[JJ Fullmer]

@gribbler Are you able to get to the fog pxe boot menu on this host?
If so can you try the Client System Information (Compatability) option?
Mainly want to see option 4 (check fog compatibility) passes (Doing this after what @Tom-Elliott suggested)

Also, if you’re building a custom ipxe.efi file to support the Marvell card, do you happen to have any other cards on this machine that you could try with the default ipxe.efi file?
Do you maybe have the FOG_KERNEL_LOGLEVEL fog setting (in fog configuration->fog settings) up higher than 4? That might explain the extra output you’re seeing

[gribbler]

@gribbler Thanks to @JJ-Fullmer and @Tom-Elliott for some direction.

We’ve just moved buildings over the weekend and I’ve not had a chance to get the proxmox servers back online, gotta get people working was the first priority. I will have a look in the next couple days and for sure no later than early next week.

Thanks again.

[gribbler]

@gribbler Move to the new building is over, had a chance today to try and get our proxmox server back online, but alas a drive was dead so I used a different server, and what may have been key is a different workstation to try to boot. I had to use the newer built ipxe.efi as pxe boots, hands off to ipxe and then it just can’t do anything. Gets an IP address but can’t see shit.

I was able to get FOG working without much hassle, I do have workstations with LVMs with XFS on them which seems to be a problem for FOG . It did say I could try to not do it as a resizable image but that I assume would then be the same size as the drive? Much too big. In doing some digging I found an option is to boot off a live CD, use partclone and I was able to clone the /home and / partitions which I can import into FOG. Further thoughts I had on the way home was that I must try to just image a partition and see if that works, and a few other ways I could try to skin the cat so to speak… It seems like I have a few directions I could head, so any advice you have that would be appreciated and might save me some trial and error.

I am pretty sure there was something up with that workstation I was trying before, I will try to image it again sometime next week I hope.

Thank you to @george1421 @JJ-Fullmer @Tom-Elliott for your guidance. It is really appreciated.

[gribbler]

Biggest thing I found from this for future searches of answers…

F*CK UEFI. Legacy FTW.