FOG Project Image Capture on Raspberry Pi 4 (ARM64) via U-Boot

Tom Elliott

@Jeremy FOS apparently hasn’t been great with Raspberry Pi.

I’m working on building in proper support and fixing a bug we found along the way (Yes I’m using AI to help with the coding and finding/implementing piece.)

I’ll need you, once this refactoring is done and an experimental build is built to test that kernel/init pair to be able to know if what we added/supposedly fixed is working.

Thank you for letting us know.

Jeremy

@Tom-Elliott

Thanks a lot for your feedback; I’ve been trying to get it working for a week without success, and your reply confirms that the issue isn’t on my end.

I’d be more than happy to help once you’ve found the solution.

Tom Elliott

@Jeremy The build is up, and it turned out there were two separate bugs, not one.

Bug 1: our arm64 kernel couldn’t unpack our own arm64 init. The kernel had gzip
initramfs support switched off while the init we publish is arm_init.cpio.gz. I
checked a released arm_Image with extract-ikconfig to be sure and it’s real. That
pair has been unable to boot since January 2021, when a cleanup commit pruned “unneeded
initrd compression algos” from all three kernel configs. Correct for x86, wrong for arm64,
and nobody caught it because almost nobody boots the arm64 build.

Bug 2: our arm64 kernel didn’t describe an ARM platform at all. No CONFIG_ARCH_BCM,
so no Broadcom SoC and no Raspberry Pi anything. That’s the missing NIC you found. It also
had no PL011 UART and no generic PCI host bridge, so it had no serial console and no PCI
bus on basically any arm64 machine. The config was added in 2018 by someone who said in
the commit message that they couldn’t test it, and eight years of make oldconfig carried
it forward untouched.

Both are fixed and merged. The kernel now has BCM2835/2711/2712 support, which covers Pi 3
through Pi 5: the VideoCore firmware and mailbox chain, the SD card controllers, bcmgenet,
lan78xx and smsc95xx for the older boards, macb for the Pi 5’s RP1, PCIE_BRCMSTB, and the
SoC watchdog (FOS reboots through that at the end of a task, so without it a client
finishes and just sits there).

I’ve tested it under QEMU as far as QEMU can go: the new kernel unpacks the released init
and boots all the way into FOS userspace, and with a virtio NIC it gets an IP. The old
kernel produces no console output at all on the same machine. What QEMU can’t test is
any of the Broadcom code, because its Pi 4 model disables the PCIe, genet, RNG and thermal
blocks. That’s the part I need you for. None of us has a Pi.

Testing it

Good news: don’t change your setup. Keep the U-Boot chain you already have working. We’re
not asking you to switch to iPXE yet, I just want to know whether the kernel runs on real
Pi hardware.

1. Get the files

# the new kernel
wget https://github.com/FOGProject/fos/releases/download/EXP_20260825-141730/arm_Image

# the init - unchanged, but grab this exact one, it's the pair I tested
wget https://github.com/FOGProject/fos/releases/download/EXP_20260819-141018/arm_init.cpio.gz

# device trees, if you'd rather use ours than the one you have
wget https://github.com/FOGProject/fos/releases/download/EXP_20260825-141730/arm_dtbs.tar.gz
tar xzf arm_dtbs.tar.gz          # gives you broadcom/bcm2711-rpi-4-b.dtb

Drop arm_Image and arm_init.cpio.gz into your 7efcf632/ TFTP directory. Your existing
DTB is fine, ours is just there if you want it.

2. Skip the uInitrd wrapper

You can hand the gzip file straight to booti and skip mkimage entirely:

tftp ${ramdisk_addr_r} 7efcf632/arm_init.cpio.gz
booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

I’d rather you did it this way for this test. If you wrap it with mkimage -C gzip,
U-Boot may decompress it before the kernel sees it, and then we won’t have tested the
decompression fix at all. If you’d rather keep uInitrd, build it with -C none so
the gzip bytes get passed through untouched.

3. Move your load addresses

This one matters. Your current layout puts the DTB at 0x02600000, which is 38 MB in,
and the kernel at 0x00080000. The new arm_Image is 30 MB and its BSS runs past the end
of the file, so it can land on top of the DTB and you’d get a hang that looks exactly like
the one you’re already seeing. Give it room:

setenv kernel_addr_r  0x00080000
setenv fdt_addr_r     0x08000000
setenv ramdisk_addr_r 0x09000000

You can drop kernel_comp_addr_r and kernel_comp_size. Those are for a compressed
kernel and arm_Image isn’t one.

4. Fix the bootargs

There are four real mistakes in what you have, and one addition that will save you a lot of
time:

storagedev=/dev/sda isn’t a thing. FOS ignores it. The variable that picks a disk
is fdrive=. That’s the “Host Primary Disk” field in the web UI.
storage= is pointing at the wrong share. A capture writes to
192.168.203.113:/images/dev/. /images is exported read-only, /images/dev is the
writable one, so as written the upload will fail on permissions after a long wait.
web= needs the scheme: web=http://192.168.203.113/fog/. curl usually guesses,
don’t rely on it.
mode=debug does nothing here. FOS only reads mode= when type= is empty, so with
type=up it’s ignored. The one you want is isdebug=yes, which drops you to a shell
and pauses after every step. Use it for this first test, it’ll show you exactly where
things stop.

So:

setenv bootargs console=tty1 console=ttyS0,115200 loglevel=7 consoleblank=0 \
  isdebug=yes \
  web=http://192.168.203.113/fog/ \
  mac=dc:a6:32:c8:b4:33 osid=50 type=up \
  img=ImagePi4 imgType=mps imgPartitionType=all \
  storage=192.168.203.113:/images/dev/ storageip=192.168.203.113 \
  fdrive=/dev/sda

(console=ttyS0 is the Pi’s mini-UART. That driver is new in this build too, so if you have
a serial adapter you’ll now get output on it. Harmless if you don’t.)

5. What success looks like

Early on you should see, before anything FOG-specific:

Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 52576K
Run /init as init process

Those three lines alone tell me bug 1 is fixed on real hardware. Then the FOG banner with
Version / Init Version / Kernel Version, then Checking Operating System ... Linux, then
Attempting to check in.

If it sits on “Attempting to check in” repeating, that is a pass, not a failure. It means
everything booted and FOS is waiting for the server to hand it a job.

Then the actual capture

FOS won’t image anything until the server has a task queued for it. Hand-writing kernel
arguments doesn’t create one, which is the other reason you weren’t getting anywhere.

In the web UI, register the Pi as a host with MAC dc:a6:32:c8:b4:33.
Set that host’s Host Primary Disk to /dev/sda (that’s what fdrive= above is
doing by hand).
Create an image called ImagePi4, Linux, Multiple Partition Image - Single Disk.
On the host, Basic Tasks -> Capture.
Boot the Pi again with the same bootargs. Check-in should return and it should start
uploading.

Fair warning on one thing I can’t predict: your Pi boots off that USB SSD, and FOS is going
to be capturing the same disk it isn’t running from, which is fine. But a Pi’s boot
partition and its partition layout aren’t like a PC’s, and I don’t know yet how our
partition code handles it. That’s the second thing I want to learn from you.

What to send back

Even a failure is useful, honestly more useful than a success.

The console output from boot, as much as you can capture. Serial is easier than
photographing a screen if you have an adapter.
Specifically whether you see those three initramfs lines.
If it stops somewhere, whatever the last line on screen is. With isdebug=yes it’ll be
paused at a named step rather than just hanging, which tells me exactly where to look.

Thanks for sticking with this. Two people have asked for Pi support on here over the last
couple of years and both threads died because none of the developers has the hardware, so
you finding this and being willing to test it is genuinely how it gets fixed.

Jeremy

@Tom-Elliott Thanks for the feedback; I started the tests and got this:

Bugs 1 & 2 are 100% fixed!

The kernel successfully unpacks the initramfs (Trying to unpack rootfs image as initramfs..., Run /init as init process).
Broadcom SoC drivers, USB storage (/dev/sda - JMicron), and the internal NIC (bcmgenet -> eth0 Link is Up) are all working great.

Where it stops:
After getting the network link up, FOS fails when trying to mount with the following errors on screen:

ext3: Unknown parameter ‘nolock’
ext2: Unknown parameter ‘nolock’
ext4: Unknown parameter ‘nolock’
vfat: Unknown parameter ‘nolock’
msdos: Unknown parameter ‘nolock’
f2fs: Unknown parameter ‘nolock’

It looks like FOS is passing the NFS nolock mount option when trying to probe/mount local filesystem types or local partitions. After these errors, the Pi reboots back to the U-Boot prompt.

Let me know if you need me to test a updated init containing a fix for the mount options!

Tom Elliott

@Jeremy That mount error is our bug. FOS was passing NFS mount options without
actually telling mount it was NFS, so when the storage path doesn’t come
through, busybox falls back to trying every local filesystem in turn and each
one rejects nolock — the ext2/ext4/vfat lines were never really involved.
Fixed and merged, and there’s a new experimental build below with it in. Test it
when you get a chance and let me know how it goes.

Two things when you do. Add isdebug=yes to your bootargs — mode=debug isn’t
our debug switch, so right now you’re getting rebooted back to U-Boot before you
can read anything. And send me the output of these from the failing boot:

uname -a
cat /proc/cmdline
cat /proc/filesystems

Your error listed f2fs, and our arm64 kernel is built without f2fs and without
module support, so it can’t have produced that. I think that boot was running
your kernel8.img with our init rather than our kernel — which would mean we
still haven’t actually tested the kernel fixes on real hardware.

wget https://github.com/FOGProject/fos/releases/download/EXP_20260826-143004/arm_Image
wget https://github.com/FOGProject/fos/releases/download/EXP_20260826-143004/arm_init.cpio.gz
wget https://github.com/FOGProject/fos/releases/download/EXP_20260826-143004/arm_dtbs.tar.gz

All three are from the same build, so there is no mixing them up with what you
have now.

Jeremy

@Tom-Elliott

Hi! I downloaded the full updated pair from build EXP_20260826-143004 (arm_Image, arm_init.cpio.gz, and bcm2711-rpi-4-b.dtb from arm_dtbs.tar.gz).

I set the memory addresses exactly as requested:

kernel_addr_r: 0x00080000
fdt_addr_r: 0x08000000
ramdisk_addr_r: 0x09000000

I also passed isdebug=yes in bootargs and removed the old kernel8.img / uInitrd references to ensure only the new build files are booted.

Result on real hardware (Pi 4B):
The Pi hangs indefinitely on the 4 Raspberry logos screen with a blinking cursor, right after U-Boot executes booti. No text output appears after the logos, and I am unable to type commands or reach the FOS shell, so I cannot run uname -a or cat /proc/....

It seems the kernel halts before initializing the console/framebuffer or hangs during early memory setup on real Broadcom hardware.

Let me know if you want me to test another build or change specific earlycon/console bootargs!

Tom Elliott

@Jeremy I found it, and it’s our bug. Our kernel is built without DRM, so it has
no vc4 driver and it depends on the firmware handing it a framebuffer the way
EFI does on a PC. On a Pi that doesn’t hold - our bcm2711-rpi-4-b.dtb has no
simple-framebuffer node, and the one the firmware would normally inject is
suppressed as soon as config.txt turns on the vc4-kms-v3d overlay, which current
Pi OS ships enabled. So our kernel had no way to write to your screen at all,
and a working boot and a dead one look exactly the same. That’s also why your
earlier test had output - kernel8.img has vc4 built in.

I’ve built the vc4 driver into our kernel. New build, same three files as
before, all from the one build:

wget https://github.com/FOGProject/fos/releases/download/EXP_20260827-114033/arm_Image
wget https://github.com/FOGProject/fos/releases/download/EXP_20260827-114033/arm_init.cpio.gz
wget https://github.com/FOGProject/fos/releases/download/EXP_20260827-114033/arm_dtbs.tar.gz

Boot it exactly the way you did last time - same load addresses, same booti
line, our DTB is fine now. One firmware note: the vc4 driver wants
avoid_warnings=2 in config.txt, otherwise the firmware can scribble over the
display setup.

All I need to know is whether you get text on the screen this time. If you do,
we’re past this and I’ll want the uname -a / cat /proc/cmdline / cat
/proc/filesystems output I asked for before. If it’s still blank, then it isn’t
the display and I’ll need serial to see what’s happening - but let’s find out
which it is first.

Jeremy

This post is deleted!

Jeremy

@Jeremy said in FOG Project Image Capture on Raspberry Pi 4 (ARM64) via U-Boot:

@Tom-Elliott

GREAT NEWS! The capture completed successfully!

Here is a summary of the steps and final configuration that made it work seamlessly on the Raspberry Pi 4:

Display Fix (Kernel side):
Downloading build EXP_20260827-114033 with the integrated VC4 driver completely resolved the display freeze on the 4 raspberries logo. HDMI output initialized perfectly.

Ramdisk Format Fix (U-Boot side):
Passing raw arm_init.cpio.gz via booti was triggering a “Wrong Ramdisk Image Format” error in U-Boot. Wrapping it into uInitrd with the following command solved the issue:
mkimage -A arm64 -O linux -T ramdisk -C none -d arm_init.cpio.gz uInitrd

Final Working boot.cmd:

setenv kernel_addr_r 0x00080000
setenv fdt_addr_r 0x08000000
setenv ramdisk_addr_r 0x09000000

setenv bootargs console=tty1 loglevel=7 consoleblank=0 web=http://192.168.203.113/fog/ mac=dc:a6:32:c8:b4:33 osid=50 type=up img=ImagePi4 imgType=mps imgPartitionType=all storage=192.168.203.113:/images/dev/ storageip=192.168.203.113 fdrive=/dev/sda

tftp ${kernel_addr_r} 7efcf632/arm_Image
tftp ${ramdisk_addr_r} 7efcf632/uInitrd
tftp ${fdt_addr_r} 7efcf632/bcm2711-rpi-4-b.dtb

booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}

Imaging Status:
Once booted into FOS, Partclone captured /dev/sda without any error and uploaded the full image to the FOG server!

Thank you so much for adding the VC4 driver and supporting ARM64 builds!

Tom Elliott

@Jeremy That’s the first time anyone has captured an image from a Raspberry Pi
with FOG, so thank you for sticking with it through three rounds of this. The
vc4 change is merged and will be in the next build.

One correction to your boot.cmd before someone copies it: you didn’t actually
need the uInitrd wrapper. The “Wrong Ramdisk Image Format” error came from
dropping the size off the ramdisk argument - without it U-Boot reads the address
as a legacy uImage and insists on a header. Either of these works:

booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

or the wrapper you built, which is fine as-is because you used -C none. That
part matters: -C gzip would have U-Boot decompress it and the kernel’s own gzip
support - the first bug we fixed in this thread - would never have been
exercised.

For anyone finding this later, the display problem was ours. We build without
DRM everywhere and rely on the firmware handing us a framebuffer, which is safe
on a PC because UEFI always publishes one. On a Pi it isn’t: our device tree has
no simple-framebuffer node, and the one the firmware would normally inject
disappears as soon as config.txt enables vc4-kms-v3d, which current Pi OS ships
turned on. So the kernel was booting fine the whole time and had no way to tell
anyone.

Worth saying plainly: your report exercised three separate fixes on real
hardware for the first time - the initramfs decompression, the ARM platform
support, and the NFS mount options. All of it had only ever run in QEMU before
you.