Raspberry Pi 4 unable to pxe boot

george1421

I’ve been working on natively compiling ipxe and FOS Linux to run on the raspberry pi 4. I’ve ran into a few issues with FOG.

First of all the raspberry pi does not support pxe booting out of the box (not a FOG issue), but the ipxe developers have a fork that supports pxe booting from a sd card https://github.com/ipxe/pipxe merging that project with FOG’s ipxe settings produces a sd card that will pxe boot into the FOG iPXE menu. Initially only did it to prove I could make it work. I just found out that the Pi 4 compute module (dedicated processor module) has an onboard MMC memory chip that can be programmed with an operating system by a convoluted process. This would be a perfect use case for FOG.

Now to the problems.
When you pxe boot right into the ipxe shell ${arch} is being set correctly to arm64 this is because cpuid is something only known to the ia32 and ia64 platforms. On arm64 the command fails and is unknown

ref: default.ipxe

cpuid --ext 29 && set arch x86_64 || set arch ${buildarch}

So arch is being set by the ${buildarch} ipxe command. For the raspberry pi build ${buildarch} is arm64 (the version I build of iPXE)

The first problem is that when you select hardware compatibility test from the FOG iPXE menu it attempts to load bzImage32. Inspecting the ${arch} value after the failure shows it being set to i386 by bootmenu.class.php

Looking at bootmenu.class.php I see a reference to arm at line 146. https://github.com/FOGProject/fogproject/blob/171d63724131c396029992730660497d48410842/packages/web/lib/fog/bootmenu.class.php#L146 IMO this test should be for arm64 or arm32 or (I think) aa64 not arm.

Edit: Looking at the linux kernel build I see the arch options are arm for 32 bit and arm64 for 64 bit.

I think I see where i386 is getting returned to ipxe because of the error in line 517

cpuid --ext 29 && set arch x86_64 || set arch i386

https://github.com/FOGProject/fogproject/blob/171d63724131c396029992730660497d48410842/packages/web/lib/fog/bootmenu.class.php#L517

That set arch i386 should be changed to match what default.ipxe has to offer. set arch ${buildarch}

The same at line 542, 699, 722, 745, 796, 821, and 1905

Fixing those lines still causes the hardware checker boot to fail with transferring bzImage32 to the target computer, but ${arch} now reports arm64 after the failure.

Looking into this more, at line 287 https://github.com/FOGProject/fogproject/blob/171d63724131c396029992730660497d48410842/packages/web/lib/fog/bootmenu.class.php#L287 it appears to only check for x64 arch and returns bzImage32 if the x64 check fails.

        if ($_REQUEST['arch'] != 'x86_64') {
            $bzImage = $bzImage32;
            $imagefile = $init_32;
        }

The code could be changed to this (I know it has hardcoded values, adding fields would also require webui changes. Until official support for ARM64 is added to FOG, this will work.

        if ($_REQUEST['arch'] != 'x86_64') {
            if ($_REQUEST['arch'] = 'arm64') {
                $bzImage = "ImagePiARM64";
                $imagefile = "init_arm64.cpio";
            } else {
                $bzImage = $bzImage32;
                $imagefile = $init_32;
            }
        }

Note: compiling the 5.9.8 version of linux using the configuration settings for 5.6.18 produces a linux kernel file named Image.gz and not bzImage. I’m still looking into this, but one step at a time.

george1421

@george1421 I’ve been able to boot into ipxe now, register the system with the fog server and capture an image from a raspberry pi. I have not tried to deploy an image to the pi because I’m using the sd card to boot the pi into ipxe and if it doesn’t deploy correctly I’ll have a broken boot device. I’m working on some other options right now, but basically FOG is imaging a raspberry pi running an ARM processor.

I need to look into this a bit more, but when partclone ran the background was RED and not normally BLUE. I don’t know what this means but that is the only thing out of normal I found so far. The rest of the imaging environment seemed. Normal. The FOG iPXE menu screen displayed the background image normally and the text was colored appropriately. So the RED in partclone is the only abnormal thing I found so far.

I did have to build a custom FOS Kernel for this Pi because the standard linux kernel could not see the USB keyboard, but using the Pi version of the linux source code did. I also had to build a ARM version of init.xz with no changes except one.

For both bzImage and init.xz I needed to remove the compression on the files. For bzImage the bzip compression is only an x86 feature. For other architectures you need to use gzip to compress the kernel. The other thing, especially for the PIs, is that they are somewhat CPU bound so this dynamic compression takes longer to download and run than just transferring a larger uncompressed image and running it. So for both the kernel and initrd.img I’m not compressing the images. For the initrd.img I’m sending it in cpio format to the Pi.

Development naming convention I’m using is
kernel: ImagePiARM64
initrd: init_arm64.cpio

The kernel is unique to the Raspberry Pi and not intended for any other ARM architecture.

For FOG 1.6.x we probably should add 2 new fields in the FOG Configuration -> FOG settings for the ARM kernels too just like we have for bzImage and bzImage32. But in this case ARM64 and ARM32. I can see one fog server imaging both x86 and ARM processors so we will need to make accommodations. I’m not sure how to handle the kernel field in the host configuration manager though.

Jacques-Olivier

@george1421

Hi there,

I am interrested of this possibility to install and clone Raspberry PI with FOG.

Did you achieve this project ?

george1421

@jacques-olivier Yes and No. I was able to unfortunately reimage my usb boot device with FOG a few times.

The short answer is I have not yet tested writting to the Pi4 Compute modules eMMC drive. In theory it should work.

This is going to be a bit of TLDR…
A little bit more behind the story is that I was verifying if fog was ready for non-intel hardware. I found several spots in the code where an intel process is assumed. I did work through them and I think FOG is ready to test imaging.

I was focusing on the Pi4 compute module with onboard emmc memory. At the time I stopped posting here, the compute module was back ordered until after Jan 1 2020. I have not looked to see if they are still unavailable. So the project got tabled.

I only see FOG imaging useful for the Pi4 compute module since there are other ways to clone the micro sd card cleaner and faster than FOG.

The Pis are an interesting breed in that the GPU boot loads the main CPU. I was able to PXE boot because the GPU was able to load what it needed from the USB stick then it chained to iPXE for imaging. I understand there is firmware that can make the Pi4 and Pi4 capable of PXE booting, but I did not want to breakdown my Pi4 to test this feature since I was using it to recompile the FOS Linux kernel.

I know its a bit more than you asked, but that is where I’m at on the project.

Jacques-Olivier

@george1421 Really thank you @george1421 for this long answer.
I think i will try another way to autoinstall my PIs until FOG pi better integration.

george1421

@jacques-olivier said in Raspberry Pi 4 unable to pxe boot:

I think i will try another way to autoinstall my PIs until FOG pi better integration.

Will you explain your problem with creating your PIs image? What version of PIs are you using? I’m trying to understand your reasons for using FOG to image these devices. For myself I only see one reason to use FOG and that is the PI4 compute module. I might be missing something so I want to understand your thoughts.

Jacques-Olivier

@george1421 I am looking for a way to easely clone (for backup) the data on several PIs

Raspberry Pi 4 unable to pxe boot

186

11.6k

17.1k

154.5k