PXE Boot Error - "Kernel Panic" Issue "XZ decompressor ran out of memory"

[lperoma]

I hope you’re doing well. I’m seeking help with a PXE boot issue I’m experiencing. The system is encountering a “Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)” error, along with an “XZ decompressor ran out of memory” message, causing the boot process to fail.

The only troubleshooting step I’ve taken so far is updating to the latest development kernel, but the problem persists. I would greatly appreciate any guidance or suggestions you may have for resolving this issue, especially if there are specific configurations, settings, or other steps I should consider.

[lperoma]

@Tom-Elliott Updating now to latest dev-branch and will report next week.

[lperoma]

I should have said I am running 1.5.10.1662 on debian 12.

Before updating to this version, I was using Debian 11 with 1.5.10.10 and had no problems.
Thanks,

[lperoma]

Switched to Kernel is 6.1.89 to avoid VM booting problem as per

https://forums.fogproject.org/topic/17663/capture-uefi-image-on-hyper-v-vm/2

Solved the VM issue but not the main one in this thread.

[lperoma]

I am bit desperate here, have been trying different Kernels and initrd images, what would happen here ?

I was working perfectly win previous versions.

[george1421]

@lperoma Looking at your screen shot a few things jump out at me.

Why is iPXE loading the 32 bit version of FOS Linux?

What hardware are you running here? What is the CPU? Is this super old hardware, like from the Pentium era?

I translate this error to the kernel (bzImage32) doesn’t have enough RAM memory to expand the FOS linux virtual hard drive (init_32.xz) into memory. If this IS a 32 bit system I still can’t understand why there is not enough ram for this kernel expansion. 32 bit arch is limited to 2GB of RAM. The FOS linux kernel and initrd compressed is < 400MB. Something is not adding up.

[lperoma]

@george1421 I have no idea. This is a 8yr old dell AIO machine, 64bit.

Before, we only had BIOS machines. No DHPCproxy, just straight into undionly.kpxe, never had a problem with BIOS machines.

Lately I have also put in place a dnsmasq service to allow for UEFI and BIOS, This are the settings:

I am not sure if it has anything to do with this, I don’t really understand the PXE process, despite investing a bit of time on it.

This are the logs from dnsmasq

3722788871 available DHCP subnet: 192.168.0.0/255.255.252.0
3722788871 vendor class: PXEClient:Arch:00000:UNDI:002001
3722788871 PXE(eth0) 5c:f9:dd:e5:40:07 proxy
3722788871 tags: BIOS, eth0
3722788871 bootfile name: ipxe.kkpxe
3722788871 next server: 192.168.0.21
3722788871 broadcast response
3722788871 sent size: 1 option: 53 message-type 2
3722788871 sent size: 4 option: 54 server-identifier 192.168.0.40
3722788871 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
3722788871 sent size: 17 option: 97 client-machine-id 00:44:45:4c:4c:44:00:10:54:80:39:b7:c0:4f…
3722788871 sent size: 50 option: 43 vendor-encap 06:01:03:08:07:80:00:01:c0:a8:00:28:09:0e…
3722788871 available DHCP subnet: 192.168.0.0/255.255.252.0
3722788871 vendor class: PXEClient:Arch:00000:UNDI:002001
3722788871 available DHCP subnet: 192.168.0.0/255.255.252.0
3722788871 vendor class: PXEClient:Arch:00000:UNDI:002001
3722788871 PXE(eth0) 192.168.2.236 5c:f9:dd:e5:40:07 ipxe.kkpxe
3722788871 tags: BIOS, eth0
3722788871 bootfile name: ipxe.kkpxe
3722788871 next server: 192.168.0.40
3722788871 sent size: 1 option: 53 message-type 5
3722788871 sent size: 4 option: 54 server-identifier 192.168.0.40
3722788871 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
3722788871 sent size: 17 option: 97 client-machine-id 00:44:45:4c:4c:44:00:10:54:80:39:b7:c0:4f…
3722788871 sent size: 28 option: 43 vendor-encap 47:04:80:00:00:00:0a:13:01:42:6f:6f:74:69…
error 0 TFTP Aborted received from 192.168.2.236
sent /tftpboot/ipxe.kkpxe to 192.168.2.236
sent /tftpboot/ipxe.kkpxe to 192.168.2.236
3942607374 available DHCP subnet: 192.168.0.0/255.255.252.0
3942607374 vendor class: PXEClient:Arch:00000:UNDI:002001
3942607374 user class: iPXE
3942607374 PXE(eth0) 5c:f9:dd:e5:40:07 proxy
3942607374 tags: BIOS, eth0
3942607374 bootfile name: ipxe.kkpxe
3942607374 next server: 192.168.0.21
3942607374 broadcast response
3942607374 sent size: 1 option: 53 message-type 2
3942607374 sent size: 4 option: 54 server-identifier 192.168.0.40
3942607374 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74
3942607374 sent size: 17 option: 97 client-machine-id 00:44:45:4c:4c:44:00:10:54:80:39:b7:c0:4f…
3942607374 sent size: 50 option: 43 vendor-encap 06:01:03:08:07:80:00:01:c0:a8:00:28:09:0e…
3942607374 available DHCP subnet: 192.168.0.0/255.255.252.0
3942607374 vendor class: PXEClient:Arch:00000:UNDI:002001
3942607374 user class: iPXE
2670414323 available DHCP subnet: 192.168.0.0/255.255.252.0
2670414323 vendor class: MSFT 5.0
2670414323 client provides name: PCAINF03.mad.mater.int
2203613339 available DHCP subnet: 192.168.0.0/255.255.252.0
2203613339 vendor class: MSFT 5.0
2203613339 client provides name: PCAINF03.mad.mater.int
4191863835 available DHCP subnet: 192.168.0.0/255.255.252.0

Am I doing something wrong ?

[george1421]

@lperoma I don’t feel this is a pxe booting issue. There is only two flavors of iPXE. One for bios and one for uefi. Is this computer in bios or uefi mode?

The only way I can see this messing up because of dhcp is that the computer is in uefi mode, and you had something wrong with dnsmasq, where its loading the 32bit version of iPXE. That might falsely miss identify the processor as a 32 bit causing bzImage32 to be called.

The other thing I can think in the fog configuration -> fog settings. If you hit the expand all button then search for bzImage. If by chance someone changed the 64 bit fields to load bzImage32 and the 32 bit initrd. But that’s really unlikely.

Do all other computers work correctly except for this one?

[lperoma]

@george1421 All these Dell AIO PCs have this failure in fairness. BIOS mode. I guess it has to do something with dnspasq missidentifiying the computer ?

Fog settings appear correctly. I don’t have and will never have 32 bit clients, so I guess a workaround could be sending 64 bzimage and initrd files to these clients identified as 32 in the setting you suggest.

I will try to troubleshoot dnsmasq and see.

Any other idea? Can some one identify any issue with my dnsmasq ?

Thanks,

[george1421]

@lperoma said in PXE Boot Error - "Kernel Panic" Issue "XZ decompressor ran out of memory":

Any other idea? Can some one identify any issue with my dnsmasq ?

Its not a dnsmasq issue. dnsmasq only instructs to pickup ipxe.efi or undionly.kpxe once that boot loader is running dnsmasq has done its job. For some reason ipxe is misidentifying the processor as 32 bit only. Will you collect the exact processor model in these AIO?

I just saw in your previous message ipxe.kkpxe is being sent. Update your dnsmasq to send undionly.kpxe instead. That shouldn’t matter because ipxe.kkpxe uses the internal network adapters and undionly.kpxe uses the generic UNDI driver. But for consistency sake, lets make sure its not an ipxe issue.

[lperoma]

@george1421 I can confirm my workaround worked, and sending 64bit kernel and initrd images boots FOS no problem.

From fog inventory:

System Manufacturer Dell Inc.
System Product OptiPlex 9020 AIO
System Version 01
System Serial Number 7DT9322
System UUID 4c4c4544-0044-5410-8039-b7c04f333232
System Type Type: All In One
BIOS Vendor Dell Inc.
BIOS Version A08
BIOS Date 05/15/2014
Motherboard Manufacturer Dell Inc.
Motherboard Product Name 0GXX2X
Motherboard Version A00
Motherboard Serial Number /7DT9322/CN7443133R001K/
Motherboard Asset Tag Not Specified
CPU Manufacturer Intel
CPU Version Intel Core i5-4570S CPU @ 2.90GHz
CPU Normal Speed Current Speed: 2900 MHz
CPU Max Speed Max Speed: 3000 MHz
Memory 7.68 GiB
Hard Disk Model KINGSTON SA400S37240G
Hard Disk Firmware SAJ20104
Hard Disk Serial Number 50026B7685C378B6
Chassis Manufacturer Dell Inc.
Chassis Version
Chassis Serial 7DT9322
Chassis Asset Not Specified
GPU-0 Vendor Advanced Micro Devices
GPU-0 Product Inc. [AMD/ATI]
GPU-1 Vendor Intel Corporation
GPU-1 Product Mars [Radeon HD 8670A/8670M/8750M / R7 M370]

Regards,

[lperoma]

@george1421 I was tenting other pxe images. It is using undionly.kpxe to BIOS machines and snp.efi for UEFI, unless you recommend anything else, I really have no idea which one to use, just using what seems most popular.

[george1421]

@lperoma Undionly.kpxe and snp.efi are probably the safest choices. Both of these use the built in driver built into the network adapter. The ipxe.pxe and ipxe.efi are more like the linux kernel where it has every known driver built into the boot loader. All of the other ipxe boot loaders are for unique situations that are not common.

So your system is an old 4th gen i5 processor. That’s strange that ipxe is misidentifying that CPU.

[lperoma]

@george1421 As FYI, this machine is also missindentified in FOG as 32 bits. The same workaround is workking fine now:

System Manufacturer Hewlett-Packard
System Product HP EliteBook 8470p
System Version A1029D1102
System Serial Number CNU336BLTL
System UUID 1e371eb7-7334-11e3-bfb9-d9e6c803c050
System Type Type: Notebook
BIOS Vendor Hewlett-Packard
BIOS Version 68ICF Ver. F.43
BIOS Date 07/16/2013
Motherboard Manufacturer Hewlett-Packard
Motherboard Product Name 179B
Motherboard Version KBC Version 42.36
Motherboard Serial Number PCWHK001X5FPRO
Motherboard Asset Tag Not Specified
CPU Manufacturer Intel Corporation
CPU Version Intel Core i5-3380M CPU @ 2.90GHz
CPU Normal Speed Current Speed: 2900 MHz
CPU Max Speed Max Speed: 4000 MHz
Memory 7.63 GiB
Hard Disk Model MTFDDAK256MAM-1K12
Hard Disk Firmware 08TH
Hard Disk Serial Number 13330385921A
Chassis Manufacturer Hewlett-Packard
Chassis Version
Chassis Serial CNU336BLTL
Chassis Asset CNU336BLTL
GPU-0 Vendor Intel Corporation
GPU-0 Product 3rd Gen Core processor Graphics Controller

[fabritrento]

This post is deleted!

[fabritrento]

This post is deleted!

[fabritrento]

@lperoma said in PXE Boot Error - "Kernel Panic" Issue "XZ decompressor ran out of memory":

I should have said I am running 1.5.10.1662 on debian 12.

Before updating to this version, I was using Debian 11 with 1.5.10.10 and had no problems.
Thanks,

seems that i have same problem on old pc. that problem don’t exist with 1.5.10 version:
https://forums.fogproject.org/topic/17699/lenovo-thinkcentre-m900-kernel-panic-with-latest-fog-1-5-10-1622-version

[Tom Elliott]

@lperoma @fabritrento Please try updating dev-branch to the latest version.

I have added some additional checking.

Granted 32 bit binaries should be able to work without problem but alas it seems there’s an issue currently.

The latest dev-branch tries to address this by taking account of the buildarch of the ipxe binary being i386, but the CPU architecture being 64 bit. I need testing please of course if both of you wouldn’t mind?

Thanks in advance.

[lperoma]

@Tom-Elliott Updating now to latest dev-branch and will report next week.

[PoeticMonk87]

I was able to find a work around after running into this issue. In FOG Settings under TFTP Server update the “TFTP PXE KERNAL 32” to bzImage and “PXE BOOT IMAGE” to init.xz. I was able get thing working after that.