Upgrade to Trunk, Kernel Panic

[SaxxAppeal]

Hi all,

I upgraded from 1.2.0 to trunk, and now I’m getting the wonderful kernel panic message when I try to load a kernel (Full Host Registration & Inventory):

“kernel panic-not syncing: VFS: unable to mount root fs on
unknown block(1,0)”

I’ve tried this on 1 physical machine and 1 virtual machine, both were 100% operating before the upgrade. I’ve tried other kernels, and older kernels from previous releases. I’ve also verified that (as far as I can tell) I have placed the correct 32 bit and 64 bit init and bzImage files in FOG settings. I’ve tried this with a Dell 745, Dell 760, and Lenovo T430.

Can anyone offer some insight into this issue? Thanks in advance for your help!

[Sebastian Roth]

Thanks @Wayne-Workman for reminding me. Added to the wiki now (see my signature). The article is still missing a lot of details about the boot process and all the issues one might run into.

[Wayne Workman]

@Sebastian-Roth said in Upgrade to Trunk, Kernel Panic:

@SaxxAppeal said:

bzImage: x86 boot sector

A while back already I noticed that the file command does not output all the valuable information on all distros. What it does is kind of simple. It reads the first couple of bytes of a file and compares it to known signatures (https://en.wikipedia.org/wiki/List_of_file_signatures). Seems like the ubuntu file command does detect the kernel as simple x86 boot sector and therefore does not print out the kernel version. Very sad because I really love that little tool. So maybe I have to get around this by pointing people to check the kernel version via the web gui - although this only works when the kernel image is in place but nor for checking the version of any bzImage you have lying around.

@george1421 said:

Strange on my prod server and dev box they both say x86, not sure if that is a type-o or what. I can see from an ls command that bzImage IS larger so that is probably the x64 image.

Although the file command is powerful (e.g. even prints the partition table if you run file d1.mbr) is does not distinguish between 32 or 64 bit kernel. I don’t know the kernel binary structure enough but I guess detecting a 64 bit bzImage from a 32 bit one is beyond the capability of a simple tool like file which just compares a few byte ranges…

Ok, back to the initial question. George is right that the kernel panic essentially means that the kernel was not able to find it’s root device (which we have packed completely into the init(_32).xz file!). This can be caused by different reasons:

• init(_32).xz missing / wrong filename - in the output you posted I see init_32.xz.1, sure you have the file named like that? Shouldn’t hurt I suppose but maybe try re-naming/-configuring it to init_32_1.xz to see if it makes a difference
• using the old pxelinux.0 for PXE booting (although I still haven’t had the time to find out why that is!) - use ipxe binaries for PXE booting, e.g. undionly.kpxe
• init(_32).xz corrupt - broken download (shouldn’t happen with FOG trunk anymore)
• 32/64 bit mixed, e.g. bzImage32 / init.xz or vice versa
• Missing initrd=init.. kernel command line parameter
• bzImage32 / init_32.xz booted on a 64 bit architecture (although this usually yields in a different error from what I know)

Possibly something else I missed!? Please add other things you know so I can add this to the wiki soon!

Bumping this post - this one is all yours, @Sebastian-Roth

[SaxxAppeal]

@SaxxAppeal Just figured it out!!

the boot line should read:

dhcp-boot=undionly.kpxe,, XXX.XXX.XXX.XXX

The double comma is required because it expects the server name first, then the IP.

Hope that helps someone…

Thanks EVERYONE for your help on this!!

[SaxxAppeal]

@Sebastian-Roth The symlink works as well… But still asks me to enter the TFTP server. I think its just a matter of knowing exactly where and how to pass the FOG IP to dnsmasq…

[Sebastian Roth]

Sure, even better symlink undionly.kpxe as undionly.0 and use:

pxe-service=X86PC, "Boot from network", undionly, x.x.x.x
dhcp-boot=undionly.kpxe, x.x.x.x, x.x.x.x

[edit] That’s another thing I hate about dnsmasq - there are options that seem to do the same kind of thing and I had to study the code to find out which is doing what - and sure I forgot again already [/edit]

[SaxxAppeal]

@Sebastian-Roth Got it, that’s what I figured. I’ve been trying to figure out where I need to pass the FOG server address, so far no luck.

Requested info:

dhcp-boot=/tftpboot/ipxe.krn

I’ve played with this line over the last week, but it seems to work great this way.

[Tom Elliott]

@Sebastian-Roth I need to add, remove the pxelinux side (unless you are requiring use of the pxelinux.0 file.)

[george1421]

While this isn’t an official document I did write a kb a while ago that gave guidance setting up dnsmasq under centos: https://forums.fogproject.org/topic/6376/install-dnsmasq-on-centos-7 So Sebastian’s recommendation for the next server in the config file is spot on.

[Sebastian Roth]

@SaxxAppeal The TFTP server (next-server option) should be send by the DHCP server (dnsmasq in your case). Maybe try adding your FOG server IP address to the pxe-service line:

pxe-service=X86PC, "Boot from network", pxelinux, x.x.x.x

As well let us know what you have for dhcp-boot= in your config.

[SaxxAppeal]

@Sebastian-Roth Tried the 745 and the Lenovo notebook, both are perfect.

Only other thing that has been happening is when I PXE boot, I get the following prompt:

“Please enter TFTP server:”

When I enter it, everything goes through fine and it brings up the FOG menu. I’m sure it’s just an argument that needs to be passed, but not sure where I need to specify this. Any ideas?

[Sebastian Roth]

Good to know. So my list of suggestions was spot on!

@SaxxAppeal Can you please try booting the Dell Optiplex 745 as well. We have another user having trouble with just that machine model!

[SaxxAppeal]

@george1421 THIS WORKED!!!

I renamed the pxelinux.0 file to something else, and created the sym link… Booted right in!

I’ll continue to test with some other machines, if anyone else has any other tidbits to contribute please let me know!

[SaxxAppeal]

@george1421 said in Upgrade to Trunk, Kernel Panic:

@SaxxAppeal Would you post the complete line from your dnsmasq for this prefix: pxe-service=X86PC

@Quazz I may have been reading more into what the OP didn’t say than reality.

The line you requested is as follows:

pxe-service=X86PC, "Boot from network", pxelinux

[george1421]

@SaxxAppeal Would you post the complete line from your dnsmasq for this prefix: pxe-service=X86PC

@Quazz I may have been reading more into what the OP didn’t say than reality.

[Quazz]

@george1421 He doesn’t need pxelinux.0, undionly.0 works fine.

[george1421]

@SaxxAppeal I don’t have a fog system in front of me, but can you confirm by the file date that pxelinux.0 has a similar date to undionly.kpxe? At one time the devs were not delivering that driver, they may have started again. If this is something supplied by dnsmasq then I can understand why things are not working.

It may also be interesting to rename this pxelinux.0 file to something and then create a symbolic link between undionly.kpxe and pxelinix.0 and then see what happens. Understand I’m just grabbing at straws here. But an old version of pxelinux.0 would create the error you are reporting because the inits would be held differently in memory.

[SaxxAppeal]

@george1421 After I installed FOG and dnsmasq, the /tftpboot directory has pxelinux.0 and pxelinux.0.old
undionly.kpxe is also present in the directory. My dnsmasq tsp.conf file points to /tftpboot as the directory to get everything from.

[george1421]

@SaxxAppeal That does add a new dimension to this issue.

For dnsmasq to work you need pxelinux.0. How did you create this? Copy undionly.kpxe or create a symbolic link to it?

I CAN see dnsmasq being an issue with uefi systems (which the dells you mentioned are bios only).

[SaxxAppeal]

Thanks for the suggestions, everyone.

In the original post I didn’t mention that I am using dnsmasq in my setup (always have), and it seems like it isn’t the issue because I get an IP from the FOG server, boot into the FOG splash screen, and am able to choose Full Host Reg/watch the bzImage and init files download to the client… Then I get the kernel panic.

Today I’ve tried setting up another virtual machine from scratch with Ubuntu 14, and just FOG with dnsmasq; same result.

Any other thoughts?