RHEL Initramfs unpacking failed: & Kernel panic - not syncing:

WT_101

Im using RHEL 9.0.0.

Issue happen booting through rEFInd
It can simulate by “PXE boot > FOG iPXE Menu > Boot From Hard DIsk > rEFInd menu then error below occur”.

However if don’t pxe, direct boot into the disk there is no error occur. I have no idea why if boot via rEFInd will having this issue. I do try to use Internal EFI shell in BIOS manually boot RHEL Grub EFI file, it have no issue booting into the OS.

Error Log

[ 0.435709] x86/cpu: SGX disabled by BIOS.
[ 6.905827] Initramfs unpacking failed: invalid magic at start of compressed archive
[ 7.341649] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 7.351090] CPU: 53 PID: 1 Comm: swapper/0 Not tainted 5.14.0-70.13.1.el9_0.x86_64 #1
[ 7.372231] Call Trace:
[ 7.375096] dump_stack_lvl+0x34/0x44
[ 7.379306] panic+0x102/0x2d4
[ 7.382846] mount_block_root+0x28c/0x29b
[ 7.387473] prepare_namespace+0x13b/0x16a
[ 7.392188] kernel_init_freeable+0x17d/0x1a2
[ 7.397197] ? rest_init+0xd0/0xd0
[ 7.401138] kernel_init+0x16/0x120
[ 7.405172] ret_from_fork+0x1f/0x30
[ 7.409427] Kernel Offset: 0xec00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 7.654445] —[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]—

hutch

We are experiencing the same issue. I’ve tried using a different VM image, reinstalling current and older versions FOG, and using older versions of ReFind. No problem at all if we try to boot from disk, but we get essentially the same (different memory addresses) error when trying to boot any Linux images. It gets to default.pxe, which hands off to ReFind. ReFind hands off to grub. Choose the kernel, and it errors out. Grub works if I boot directly to it, but going through FOG throws the error.

Tom Elliott

@hutch Have you guys tried using the Exit Type of just “EXIT”?

Refind has loaders that might be adjusting the memory space that the initramfs is trying to load into at the same time Refind exists.

Please note that the reason all these exit types exist in the first place is we have no idea of how things will work best. Some systems Exit works fine, for others refind is a better option. It’s a great ol’l guessing game and nobody knows the right answer.

hutch

@Tom-Elliott Thanks for the quick response! Setting Host EFI Exit Type to EXIT leads to a different error in HyperV:
Virtual machine Boot Summary

Network Adapter (00155D0D3110)
The operating system loaded.

The screen just hangs there with no OS loaded.

When trying to load a VM in Proxmox; EXIT produces a UEFI options screen, using ReFind produces the OP error.

WT_101

@Tom-Elliott From my end if i set to EXIT. It will return to BIOS instead of booting into the OS

WT_101

@hutch may i know which older versions of FOG you experience this issue? I experience it at 1.5.9

hutch

@WT_101 Tried going back to 1.5.7. Didn’t want to go back too far since we need newer drivers available.

WT_101

@Tom-Elliott
Have new finding, I tested the snp.efi from Fogproject 1.5.7 & 1.5.8 there is no issue booting into RHEL while from 1.5.9 and 1.5.10 having the kernel panic issue

george1421

@WT_101 said in RHEL Initramfs unpacking failed: & Kernel panic - not syncing::

snp.efi from Fogproject 1.5.7 & 1.5.8

Unless you are using SANBOOT or EXIT (exit) modes, snp.efi or iPXE for that matter is not responsible for booting a uefi computer. The responsibility for booting uefi computers is handed over to rEFInd. I might suspect that refind version changed between 1.5.8 and 1.5,10.

So do you mean to say if you take snp.efi from fog 1.5.7 or .8 and copy it over to FOG 1.5.10 install you can boot rhel 9?

If you were to recompile iPXE with the very latest code doe it boot rhel 9 correctly (assuming that iPXE is at fault here)?

Sebastian Roth

@WT_101 @hutch Looks like we have a very similar issue with Ubuntu 22.04 installs as well: https://forums.fogproject.org/topic/16850/problem-after-deployment-with-release-1-5-10

WT_101

@george1421 im not using both SANBOOT or EXIT.

Im using REFIND_EFI and yes im just replacing the snp.efi in the tftp folder getting from 1.5.8 (success boot into Redhat i just try again few min ago) and 1.5.10 (failed with kernel panic)

EXIT i already tried. It wont boot into the OS. it will just exit into the BIOS screen

hutch

@WT_101 Thanks! The old install didn’t seem to work, but I just grabbed the files directly out of the installation folder and it booted just fine.

WT_101

@george1421 sorry there are some wrong info provided 1.5.9 is working as well due to some mistakes i made causing i providing wrong info.

New finding i found was 1.5.9 snp.efi came with ipxe 1.20.1 there is no issue on booting into the RHEL OS with REFIND_EFI
while 1.5.10 snp.efi that are using ipxe 1.21.1 having issue to boot into RHEL OS

As there are only 3 file being dl into the system
1 snp.efi
2. refind.conf
3. refind_x64.efi

i perform comparison between these 3 file 1.5.9 vs 1.5.10

Is it possible that iPXE 1.21.1 require a newer version of refind_x64.efi ? Could you guide me on how can i troubleshoot further?

Tom Elliott

@WT_101 There were a lot of changes between v1.20.1 and v1.21.1 though we (FOG) don’t manage the repository of ipxe itself.

Here’s the code where I believe things might’ve broken the boot capabilities, but this is still just a best guess based on the changed code, not really an “assured” here’s what’s definitely broke.

A git bisect might be useful so we can see potentially where things are good -> bad and indicate to ipxe development so a more proper fix might be addressable.
https://github.com/ipxe/ipxe/compare/v1.20.1...v1.21.1#diff-9ec49bac9202333b51ea932869865503ace555e43797e9e26898e4a5b813944e

CedricoR

@hutch Hello, on my side, when i try tu use snp.efi from the 1.5.8 tarball, I have a permission issue on the boot.php file… But if i use original snp.efi i don’t have permission problem, but i get kernel panic message like with ipxe.efi. I try to troubleshoot this permission issue, but everything looks good, (chown -R fogproject:apache /var/www/html/fog/service/ipxe + chmod -R 775 /var/www/html/fog/service/ipxe)
Did i forget something ? Is there any other changes to make for it to work?

CedricoR

@CedricoR Hi everyone! I did not find a solution, I tried to change kernel side server and client, I also tried several versions of ipxe.efi and snp.efi (1.5.8, 1.5 .9 and 1.5.10), I increased the size KERNEL RAMDISK SIZE, same result, and I see nothing more when I launch a task in debug mode and when I launch the memtest I have the following message “could not boot, exec format error (https://ipxe…)”
I also launched wireshark, I share the .pcap with you, nothing abnormal except that I don’t see anything happening in http when it retrieves the refind.conf and refind_x64.
TEST2.pcap
Finally if I try a reinstallation with version 1.5.8, the network card is not recognized with default kernel.
So it’s either a kernel problem, or permission on boot.php or a hardware problem. I remain stuck, do you have any ideas where I could look to fix this problem?
What’s weird is that the grub launches correctly after refind is loaded. I can navigate in the menus, but as soon as I launch “ubuntu”, the kernel panic appears

thomGee

Hi there,
this Problem allready existed in this forum and has been signed solved, as the threadstarter resigned and used the EXIT-way in Fog
(https://forums.fogproject.org/topic/16762/local-grub-booting-fails-from-fog-menu-pxe-uefi/7)

For me the problem persisted, as I need the features in refind for dualbooting the systems.

Meanwhile I tried some more scenarios.
My PC’s have 2 OS’s : Windows 10 and Ubuntu

With Ubuntu 20.04 everything was fine untill one Update (still have the Image) and all version from there on show the the Kernel panic.

I tried several configs in initramfs - different compressions, I thought it is a bug

I think I followed all descriptions from all experts in the net for this Error.

But the system (as you allready know) boots fine, when booted directly

So I start to ask myself - what is the difference?

refind and booting from the memory fog-pxe presents.

is it possible, the compression gots worse?
that for some reason the initramfs gots bigger
is there a way to reduce the initramfs filesize

It is a pitty ther is not enough logging

RHEL Initramfs unpacking failed: & Kernel panic - not syncing:

Error Log

250

11.6k

17.1k

154.6k