Upgrade to Trunk, Kernel Panic

SaxxAppeal

Hi all,

I upgraded from 1.2.0 to trunk, and now I’m getting the wonderful kernel panic message when I try to load a kernel (Full Host Registration & Inventory):

“kernel panic-not syncing: VFS: unable to mount root fs on
unknown block(1,0)”

I’ve tried this on 1 physical machine and 1 virtual machine, both were 100% operating before the upgrade. I’ve tried other kernels, and older kernels from previous releases. I’ve also verified that (as far as I can tell) I have placed the correct 32 bit and 64 bit init and bzImage files in FOG settings. I’ve tried this with a Dell 745, Dell 760, and Lenovo T430.

Can anyone offer some insight into this issue? Thanks in advance for your help!

george1421

The error messages gives me the impression that the inits (virtual hard drive) did not make it to the target computer or its corrupt.

What boot file are you using? Because of the age of these systems I’ll assume its undionly.kpxe?

SaxxAppeal

Correct, I am using undionly.kpxe

george1421

ok on the fog server console I need you to navigate to

/var/www/html/fog/service/ipxe or /var/www/html/fog/service/ipxe depending on your distro and confirm that you have the bzImage (should since you are getting the sync error) and the init.xz and init32.xz files there.

Also when you are pxe booting are there any other warning messages? You should see it download bzImage and init.xz during the booting process.

SaxxAppeal

@george1421 No other warning messages, just an immediate kernel panic with the included text (after watching the bzImage and init files download)

I am on Ubuntu 14 server.

Checked the location you provided (it’s the same location twice btw), I am familiar with the directory as it is where the kernels download. All necessary files are accounted for…

george1421

on the bzImage files from that directory what do you get when you run file bzImage

I would expect it to look something like:

bzImage: Linux kernel x86 boot executable bzImage, version 4.5.3 (root@debian64) #1 SMP Mon May 9 05:44:34 EDT 2016, RO-rootFS, swap_dev 0x6, Normal VGA

It is important that the bzImage and init.xz versions match. You can use newer kerenels with older inits but you can use an old kernel with the newer inits.

SaxxAppeal

When I run “file bzImage”, my output is:

bzImage: x86 boot sector

So it seems the bzImage file in place is the wrong architecture, if I’m not mistaken??

I wonder how that happened. If I am right, how can I obtain a new x64 version?

george1421

@SaxxAppeal Ok you futz with the images/inits. Let me get the stuff you need. Give me a minute.

[edit]
Navigate to the ipxe directory again and run these commands to download the current stuff.

sudo wget https://fogproject.org/inits/init.xz
sudo wget https://fogproject.org/inits/init_32.xz
sudo wget https://fogproject.org/kernels/bzImage
sudo wget https://fogproject.org/kernels/bzImage32

Then repeat the file command.

george1421

OK, now I’m confused because I just reread my output and my says x86 boot image too. I need to check something.

[edit] Strange on my prod server and dev box they both say x86, not sure if that is a type-o or what. I can see from an ls command that bzImage IS larger so that is probably the x64 image.

[root@localhost ipxe]# file bzImage
bzImage: Linux kernel x86 boot executable bzImage, version 4.5.3 (root@debian64) #1 SMP Mon May 9 05:44:34 EDT 2016, RO-rootFS, swap_dev 0x6, Normal VGA
[root@localhost ipxe]# file bzImage32
bzImage32: Linux kernel x86 boot executable bzImage, version 4.5.3 (root@debian64) #1 SMP Mon May 9 05:45:31 EDT 2016, RO-rootFS, swap_dev 0x6, Normal VGA

SaxxAppeal

@george1421 Okay, no problem, thanks for the help.

FWIW, I ran wget and pulled fresh files. Same result when I run the file command, same result when the client tries to load the kernel.

george1421

I just checked and I can pxe boot one of my (vmware) mdt vms without issue.

SaxxAppeal

@george1421 I just tried again. Full error text:

"Kernel panic - not syncing: VFS: Unable to mount root fs on unknown block(1,0)
Kernel Offset: disabled
—[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)

george1421

@SaxxAppeal the message is saying that the init is damaged. I guess we need to wait for one of the devs to chime in here since this is a bit beyond what we’ve done so far.

SaxxAppeal

@george1421 I see. Thanks for the help so far! I hope someone else can shed some light, I’ve been pulling my hair out for 2 days with this one.

Quazz

@SaxxAppeal Is it possible to test this on yet another machine (preferably different hardware entirely) in the mean time? Just to rule out a few things.

SaxxAppeal

@Quazz I’ve tested it so far with a physical Dell 745 as the server, and a VirtualBox machine as the server.

I’ve tried with a Dell 745 client, Dell 760 client, Lenovo T430 client…

Not sure if I have any other hardware available that is vastly different from the ones I’ve mentioned.

Tom Elliott

What’s output in the browser?

SaxxAppeal

@Tom-Elliott Hi Tom!! What do you mean by browser output? The GUI loads and operates perfectly, not sure if you’re asking something else.

Tom Elliott

http://<ip>/fog/service/ipxe/boot.php?mac=<macofhost>

That’s the link you should go to (making relevant changes)

SaxxAppeal

@Tom-Elliott said in Upgrade to Trunk, Kernel Panic:

http://<ip>/fog/service/ipxe/boot.php?mac=<macofhost>

Sorry for the delay.

Here is the output:

#!ipxe
set fog-ip 10.1.0.154
set fog-webroot fog
set boot-url http://${fog-ip}/${fog-webroot}
cpuid --ext 29 && set arch x86_64 || set arch i386
goto get_console
:console_set
colour --rgb 0x00567a 1 ||
colour --rgb 0x00567a 2 ||
colour --rgb 0x00567a 4 ||
cpair --foreground 7 --background 2 2 ||
goto MENU
:alt_console
cpair --background 0 1 ||
cpair --background 1 2 ||
goto MENU
:get_console
console --picture http://10.1.0.154/fog/service/ipxe/bg.png --left 100 --right 80 && goto console_set || goto alt_console
:MENU
menu
colour --rgb 0xff0000 0 ||
cpair --foreground 1 1 ||
cpair --foreground 0 3 ||
cpair --foreground 4 4 ||
item --gap Host is NOT registered!
item --gap -- -------------------------------------
item fog.local Boot from hard disk
item fog.memtest Run Memtest86+
item fog.reginput Perform Full Host Registration and Inventory
item fog.reg Quick Registration and Inventory
item fog.quickimage Quick Image
item fog.multijoin Join Multicast Session
item fog.sysinfo Client System Information (Compatibility)
choose --default fog.local --timeout 3000 target && goto ${target}
:fog.local
sanboot --no-describe --drive 0x80 || goto MENU
:fog.memtest
kernel memdisk iso raw
initrd memtest.bin
boot || goto MENU
:fog.reginput
kernel bzImage32.1 loglevel=4 initrd=init_32.xz.1 root=/dev/ram0 rw ramdisk_size=127000 keymap= web=10.1.0.154/fog/ consoleblank=0 rootfstype=ext4 loglevel=4 mode=manreg
imgfetch init_32.xz.1
boot || goto MENU
:fog.reg
kernel bzImage32.1 loglevel=4 initrd=init_32.xz.1 root=/dev/ram0 rw ramdisk_size=127000 keymap= web=10.1.0.154/fog/ consoleblank=0 rootfstype=ext4 loglevel=4 mode=autoreg
imgfetch init_32.xz.1
boot || goto MENU
:fog.quickimage
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param qihost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:fog.multijoin
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param sessionJoin 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:fog.sysinfo
kernel bzImage32.1 loglevel=4 initrd=init_32.xz.1 root=/dev/ram0 rw ramdisk_size=127000 keymap= web=10.1.0.154/fog/ consoleblank=0 rootfstype=ext4 loglevel=4 mode=sysinfo
imgfetch init_32.xz.1
boot || goto MENU
:bootme
chain -ar http://10.1.0.154/fog/service/ipxe/boot.php##params ||
goto MENU
autoboot```

Upgrade to Trunk, Kernel Panic

158

12.0k

17.3k

155.2k