Upgrade to Trunk, Kernel Panic

[george1421]

OK, now I’m confused because I just reread my output and my says x86 boot image too. I need to check something.

[edit] Strange on my prod server and dev box they both say x86, not sure if that is a type-o or what. I can see from an ls command that bzImage IS larger so that is probably the x64 image.

[root@localhost ipxe]# file bzImage
bzImage: Linux kernel x86 boot executable bzImage, version 4.5.3 (root@debian64) #1 SMP Mon May 9 05:44:34 EDT 2016, RO-rootFS, swap_dev 0x6, Normal VGA
[root@localhost ipxe]# file bzImage32
bzImage32: Linux kernel x86 boot executable bzImage, version 4.5.3 (root@debian64) #1 SMP Mon May 9 05:45:31 EDT 2016, RO-rootFS, swap_dev 0x6, Normal VGA

[SaxxAppeal]

@george1421 Okay, no problem, thanks for the help.

FWIW, I ran wget and pulled fresh files. Same result when I run the file command, same result when the client tries to load the kernel.

[george1421]

I just checked and I can pxe boot one of my (vmware) mdt vms without issue.

[SaxxAppeal]

@george1421 I just tried again. Full error text:

"Kernel panic - not syncing: VFS: Unable to mount root fs on unknown block(1,0)
Kernel Offset: disabled
—[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)

[george1421]

@SaxxAppeal the message is saying that the init is damaged. I guess we need to wait for one of the devs to chime in here since this is a bit beyond what we’ve done so far.

[SaxxAppeal]

@george1421 I see. Thanks for the help so far! I hope someone else can shed some light, I’ve been pulling my hair out for 2 days with this one.

[Quazz]

@SaxxAppeal Is it possible to test this on yet another machine (preferably different hardware entirely) in the mean time? Just to rule out a few things.

[SaxxAppeal]

@Quazz I’ve tested it so far with a physical Dell 745 as the server, and a VirtualBox machine as the server.

I’ve tried with a Dell 745 client, Dell 760 client, Lenovo T430 client…

Not sure if I have any other hardware available that is vastly different from the ones I’ve mentioned.

[Tom Elliott]

What’s output in the browser?

[SaxxAppeal]

@Tom-Elliott Hi Tom!! What do you mean by browser output? The GUI loads and operates perfectly, not sure if you’re asking something else.

[Tom Elliott]

http://<ip>/fog/service/ipxe/boot.php?mac=<macofhost>

That’s the link you should go to (making relevant changes)

[SaxxAppeal]

@Tom-Elliott said in Upgrade to Trunk, Kernel Panic:

http://<ip>/fog/service/ipxe/boot.php?mac=<macofhost>

Sorry for the delay.

Here is the output:

#!ipxe
set fog-ip 10.1.0.154
set fog-webroot fog
set boot-url http://${fog-ip}/${fog-webroot}
cpuid --ext 29 && set arch x86_64 || set arch i386
goto get_console
:console_set
colour --rgb 0x00567a 1 ||
colour --rgb 0x00567a 2 ||
colour --rgb 0x00567a 4 ||
cpair --foreground 7 --background 2 2 ||
goto MENU
:alt_console
cpair --background 0 1 ||
cpair --background 1 2 ||
goto MENU
:get_console
console --picture http://10.1.0.154/fog/service/ipxe/bg.png --left 100 --right 80 && goto console_set || goto alt_console
:MENU
menu
colour --rgb 0xff0000 0 ||
cpair --foreground 1 1 ||
cpair --foreground 0 3 ||
cpair --foreground 4 4 ||
item --gap Host is NOT registered!
item --gap -- -------------------------------------
item fog.local Boot from hard disk
item fog.memtest Run Memtest86+
item fog.reginput Perform Full Host Registration and Inventory
item fog.reg Quick Registration and Inventory
item fog.quickimage Quick Image
item fog.multijoin Join Multicast Session
item fog.sysinfo Client System Information (Compatibility)
choose --default fog.local --timeout 3000 target && goto ${target}
:fog.local
sanboot --no-describe --drive 0x80 || goto MENU
:fog.memtest
kernel memdisk iso raw
initrd memtest.bin
boot || goto MENU
:fog.reginput
kernel bzImage32.1 loglevel=4 initrd=init_32.xz.1 root=/dev/ram0 rw ramdisk_size=127000 keymap= web=10.1.0.154/fog/ consoleblank=0 rootfstype=ext4 loglevel=4 mode=manreg
imgfetch init_32.xz.1
boot || goto MENU
:fog.reg
kernel bzImage32.1 loglevel=4 initrd=init_32.xz.1 root=/dev/ram0 rw ramdisk_size=127000 keymap= web=10.1.0.154/fog/ consoleblank=0 rootfstype=ext4 loglevel=4 mode=autoreg
imgfetch init_32.xz.1
boot || goto MENU
:fog.quickimage
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param qihost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:fog.multijoin
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param sessionJoin 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:fog.sysinfo
kernel bzImage32.1 loglevel=4 initrd=init_32.xz.1 root=/dev/ram0 rw ramdisk_size=127000 keymap= web=10.1.0.154/fog/ consoleblank=0 rootfstype=ext4 loglevel=4 mode=sysinfo
imgfetch init_32.xz.1
boot || goto MENU
:bootme
chain -ar http://10.1.0.154/fog/service/ipxe/boot.php##params ||
goto MENU
autoboot```

[george1421]

Cross link duplicate issue
https://forums.fogproject.org/topic/7566/kernel-panic-not-syncing-vfs

[Tom Elliott]

This seems, at least initially, unrelated.

I’m taking a look just to be on the safe side though. I’m probably still an idiot. Should be relatively simple to correct for.

[Tom Elliott]

So I’m currently 100% updated. 7853.

I’m doing the most complicated form of imaging (multicast cause I like to torture myself) and it booted everything, with no issues.

This is making me wonder if it could be related to UEFI booting?

[george1421]

@SaxxAppeal said:

I’ve tried with a Dell 745 client, Dell 760 client, Lenovo T430 client…

Tom, I thought that initially too. It was the handoff between ipxe and bzImage under a uefi system. But the 2 of the 3 systems the OP mentioned don’t know uefi. UEFI wasn’t supported in the OptiPlex line until 790/390 generation. These two dells are bios only.

We have seen this error before, but mainly with uefi systems, that I agree.

[Sebastian Roth]

@SaxxAppeal said:

bzImage: x86 boot sector

A while back already I noticed that the file command does not output all the valuable information on all distros. What it does is kind of simple. It reads the first couple of bytes of a file and compares it to known signatures (https://en.wikipedia.org/wiki/List_of_file_signatures). Seems like the ubuntu file command does detect the kernel as simple x86 boot sector and therefore does not print out the kernel version. Very sad because I really love that little tool. So maybe I have to get around this by pointing people to check the kernel version via the web gui - although this only works when the kernel image is in place but nor for checking the version of any bzImage you have lying around.

@george1421 said:

Strange on my prod server and dev box they both say x86, not sure if that is a type-o or what. I can see from an ls command that bzImage IS larger so that is probably the x64 image.

Although the file command is powerful (e.g. even prints the partition table if you run file d1.mbr) is does not distinguish between 32 or 64 bit kernel. I don’t know the kernel binary structure enough but I guess detecting a 64 bit bzImage from a 32 bit one is beyond the capability of a simple tool like file which just compares a few byte ranges…

Ok, back to the initial question. George is right that the kernel panic essentially means that the kernel was not able to find it’s root device (which we have packed completely into the init(_32).xz file!). This can be caused by different reasons:

• init(_32).xz missing / wrong filename - in the output you posted I see init_32.xz.1, sure you have the file named like that? Shouldn’t hurt I suppose but maybe try re-naming/-configuring it to init_32_1.xz to see if it makes a difference
• using the old pxelinux.0 for PXE booting (although I still haven’t had the time to find out why that is!) - use ipxe binaries for PXE booting, e.g. undionly.kpxe
• init(_32).xz corrupt - broken download (shouldn’t happen with FOG trunk anymore)
• 32/64 bit mixed, e.g. bzImage32 / init.xz or vice versa
• Missing initrd=init.. kernel command line parameter
• bzImage32 / init_32.xz booted on a 64 bit architecture (although this usually yields in a different error from what I know)

Possibly something else I missed!? Please add other things you know so I can add this to the wiki soon!

[SaxxAppeal]

Thanks for the suggestions, everyone.

In the original post I didn’t mention that I am using dnsmasq in my setup (always have), and it seems like it isn’t the issue because I get an IP from the FOG server, boot into the FOG splash screen, and am able to choose Full Host Reg/watch the bzImage and init files download to the client… Then I get the kernel panic.

Today I’ve tried setting up another virtual machine from scratch with Ubuntu 14, and just FOG with dnsmasq; same result.

Any other thoughts?

[george1421]

@SaxxAppeal That does add a new dimension to this issue.

For dnsmasq to work you need pxelinux.0. How did you create this? Copy undionly.kpxe or create a symbolic link to it?

I CAN see dnsmasq being an issue with uefi systems (which the dells you mentioned are bios only).

[SaxxAppeal]

@george1421 After I installed FOG and dnsmasq, the /tftpboot directory has pxelinux.0 and pxelinux.0.old
undionly.kpxe is also present in the directory. My dnsmasq tsp.conf file points to /tftpboot as the directory to get everything from.