FOG Project Image Capture on Raspberry Pi 4 (ARM64) via U-Boot

[Jeremy]

@Tom-Elliott

Thank you so much for the detailed explanation and for all your hard work on this!

It’s amazing to know that this was a first for the Raspberry Pi and FOG Project, and I’m really glad my tests helped validate those three major fixes on real hardware.

Good catch on the :filesize argument syntax for booti as well. That clarification will definitely be super helpful for anyone setting up ARM64/Pi deployments in the future!

Everything is working smoothly on my end now. Thanks again for the incredible reactivity and support.

The next test is to deploy the image to other Raspberry Pis; I’ll keep you posted.

[Jeremy]

@Tom-Elliott

I’ve been working on deploying images to a Raspberry Pi 4 (ARM64) using FOG (version 1.5.10.2254, FOS kernel 6.18.38), and I wanted to share our findings, encountered errors, and current limitations we faced during the process. Hopefully, this can help improve native ARM support or guide others trying to achieve unattended RPi deployments.

Here is a summary of what we experienced and how we bypassed it:

1. U-Boot bootargs Buffer Overflow

   Issue: When adding extra environment variables to bootargs inside boot.cmd (such as custom storage IPs, disk targets, or MAC addresses), the ARM64 kernel command line exceeded the maximum allowed size. This caused the kernel boot process to silently halt right after the network interface came up (Link is Up).

   Workaround: Kept the bootargs string as minimal as possible and relied on FOG web settings where applicable.

2. Missing osid Parameter

   Issue: FOS initialization stops with No os id passed (determineOS) and reboots if the kernel parameters do not explicitly provide osid=50 (or the corresponding OS ID) alongside type=down.

   Workaround: Explicitly appending osid=50 type=down in the U-Boot script parameters.

3. FOG Automatic Script Orchestration vs. ARM Multi-Partition Layout

   Issue: Even when successfully booting into FOS and passing the correct variables (img=Raspberry, storage=…, fdrive=/dev/sda), the automated fog script skips or fails to correctly parse multi-partition layouts on USB/SD external drives (/dev/sda), directly jumping to “Task Complete” without actually writing data via Partclone or triggering errors like Fatal Error: Unknown request type :: Null.

   Workaround: We had to enter FOG Debug Mode (isdebug=yes), export variables manually (export storage, export img, export type=down), and notice that FOG’s single-partition versus multi-partition routines expect specific raw file structures (d1.mbr, d1p1.img, d1p2.img).

4. Raw Image Files (.img) vs. Partclone Format

   Issue: When attempting manual deployment via partclone.restore, Partclone throws This is not partclone image because FOG stores standard multi-partition RPi captures as raw binary dumps (.img) rather than compressed partclone streams.

   Workaround: Restoring the MBR and partitions required direct block-level commands:
   Bash

   dd if=/images/Raspberry/d1.mbr of=/dev/sda bs=512 count=1
   dd if=/images/Raspberry/d1p1.img of=/dev/sda1 bs=4M status=progress
   dd if=/images/Raspberry/d1p2.img of=/dev/sda2 bs=4M status=progress

   Result: While the partitions are written, the RPi firmware still struggles to directly boot the resulting USB structure (Unable to read partition as FAT), indicating that standard FOG MBR generation (d1.mbr) doesn’t align cleanly with RPi’s expected GPT/FAT bootloader requirements for USB-MSD booting.

It seems the automated deployment engine (fog script) inside FOS needs specific adaptations for ARM/Raspberry Pi multi-partition disk imaging (handling fdrive=/dev/sda and raw dd-like partition layouts seamlessly).

Any insights or recommendations on how to properly handle native RPi deployments through FOG would be greatly appreciated!

[Tom Elliott]

@Jeremy Stop the dd approach - it’s what’s breaking the disk, not the Pi.

Those .img files are not raw dumps. They’re zstd-compressed partclone streams;
check any of them and you’ll see the zstd magic 28 b5 2f fd in the first four
bytes. So dd’ing d1p1.img onto /dev/sda1 writes compressed bytes straight onto
the partition, which is exactly why the firmware then can’t read it as FAT.
That’s also why partclone.restore told you it wasn’t a partclone image - it was
looking at the zstd wrapper. FOS does this at funcs.sh:847:

zstdmt -dc /images/Raspberry/d1p1.img | partclone.restore --ignore_crc -O /dev/sda1 -N -f 1

One per partition. If the image was captured with a gzip format instead, swap
zstdmt for pigz -dc.

The MBR is wrong too. d1.mbr isn’t one sector - we capture everything up to the
first partition, capped at 1MiB, so it’s usually 2048 sectors. Your count=1
threw away everything between the boot sector and the first partition. Drop the
count entirely and let it write the whole file:

dd if=/images/Raspberry/d1.mbr of=/dev/sda bs=512

Now the more useful finding. “Fatal Error: Unknown request type :: Null” is
FOS telling you $type arrived empty, and that’s the same problem as your
missing osid - your bootargs are being truncated before FOS ever reads them. It
isn’t the kernel: arm64’s COMMAND_LINE_SIZE is 2048, the same as x86, and the
bootargs line you posted is 233 characters. Look at U-Boot’s own command and
console buffer instead, that’s where the ceiling is on your setup.

Which gets to the real limitation, and I’d rather be straight about it: FOG’s
server has no U-Boot support at all. boot.php emits iPXE, and nothing generates
a boot.scr, delivers a DTB, or builds arguments for a non-iPXE client. Every
variable you’re hand-assembling is one the server would normally hand the
client, and doing it by hand is why the ordering and length problems are yours
to fight.

Two ways forward. Keep hand-rolling U-Boot, in which case use the two commands
above and keep bootargs minimal. Or put UEFI firmware on the Pi (pftf/RPi4) so
it PXE-boots into iPXE like any other machine and the server drives the whole
thing - that’s the route that already works today, and it’s where native
support would build from.

Capture works, which is the half that was broken. Deploy needs the server side
to exist.

[Tom Elliott]

@Jeremy Last time I told you FOG’s server had no U-Boot support and your two
options were to keep hand-rolling it or put UEFI firmware on the Pi. That’s no
longer true - I built the server side. It’s merged into working-1.6 as of
1.6.0-beta.4318, so update and you’ll have it.

There’s a new endpoint:

http://<fogserver>/fog/service/uboot/boot.php?mac=<the Pi's MAC>

It answers with an extlinux.conf-style document, computed live from the
database. If the host has a task queued you get kernel/initrd/append for FOS;
if it doesn’t, you get localboot and U-Boot falls through to the disk. Fetch it
with curl first - the whole thing is plain text and readable, which is the
point.

On the Pi:

dhcp
setenv pxefile_addr_r 0x02000000
wget ${pxefile_addr_r} http://<fogserver>/fog/service/uboot/boot.php?mac=${ethaddr}
pxe boot ${pxefile_addr_r}

Use pxe boot, not sysboot. Every extlinux example you’ll find online shows
sysboot, and it’s wrong here - sysboot reads a config off a filesystem on a
block device, so it can’t see what wget just put in memory and does nothing
visible. That’s the one I’d expect to catch people out.

This should also fix your truncation. “Unknown request type :: Null” was $type
arriving empty because your bootargs were being cut short, and I pointed at
U-Boot’s command buffer rather than the kernel. pxe boot doesn’t go through
that buffer - it parses the append line out of the loaded file straight into
bootargs, so the ceiling you were hitting isn’t in the path any more. Worth
confirming, since I’m reasoning about your build rather than looking at it.

Two deliberate omissions, both because they’re properties of your board and not
of FOG:

FOG serves no device tree. No fdt or fdtdir directive, so you keep the tree at
${fdtcontroladdr} - the one VideoCore just built for the exact board revision
it’s running on, which is better than anything I could serve you. Load
addresses stay yours for the same reason.

And FOG writes no pxelinux.cfg files. The endpoint is stateless: the board asks
by MAC and gets an answer out of the database, so there’s nothing on disk to
drift from what’s actually queued and nothing to clean up when a task finishes.

One thing to check before you start: if your FOG install has netboot set to
HTTPS, none of this will work. U-Boot’s wget is HTTP-only with no TLS at all,
so it can’t even fail a certificate check - the redirect just ends the boot with
nothing on screen. The installer now exempts service/uboot/ from the HTTP-to-
HTTPS redirect the same way it does service/ipxe/, but only when netboot is
configured for HTTP in the first place.

Full write-up is in docs/UBOOT_ARM_BOOT.md in the repo.

Being straight about what this is: the emitted config is pinned byte-for-byte
by tests and the decisions behind it are the same code the iPXE path runs, so I
know FOG emits what I intended. What I can’t tell you is that a real U-Boot
consumes it, because nobody here has a Pi. You’re the only person who can prove
that, and if it’s wrong I’d rather hear it than not.

[Jeremy]

This post is deleted!

[Tom Elliott]

@Jeremy you seem to be trying to use fogs boot system to do your own thing. What you’re asking is perfectly possible just not via debug.

Debug has a very specific purpose so to be trying to say: “hey can you stop doing what your program is intended to do so I can do things I want using your program?” Isn’t a good use of anyone’s time, particularly the developers.

What you want to do is already possible, but you’ll have to make your own post init script.

Seeing as what you want to do is not have a fog server, but to do something else, I cannot tell you what needs to happen.

You will need to read the documentation, lax as it may be, create a custom task (not debug) to do your work autonomously, and learn how the post init scripts work.

[Jeremy]

@Tom-Elliott

1. Context and Objective

   Hardware: Raspberry Pi (ARM architecture) running U-Boot.

   FOG Server: Updated to the working-1.6 branch (tested version: 1.6.0-beta.4707).

   Objective: Enable fully automated and dynamic deployment/boot (without human intervention or blocking debug mode) using the new U-Boot endpoint created by the developer (/fog/service/uboot/boot.php).

2. Testing Process and Results

   Step A: Initial Manual Validation (Success)

    By executing the deployment commands manually on the Raspberry Pi terminal, the image cloning completed successfully, and the image was properly copied to the new Raspberry Pi. This confirmed that the hardware and image transfer parts are functional.
   

   Step B: FOG Server Update

    Updated the server to the working-1.6 branch and executed ./bin/installfog.sh to integrate the new U-Boot endpoint.
   

   Step Testing the U-Boot Endpoint (boot.php)

    The curl request executed on the FOG server for a specific MAC address:
    Bash
   
    curl "http://127.0.0.1/fog/service/uboot/boot.php?mac=88:a2:9e:53:34:c0"
   
    Server Response: The endpoint correctly generates and returns a script in PXE/Syslinux format structured as follows:
    Plaintext
   
    # Generated by FOG Project. Do not edit -- it is not stored.
    default fog
    timeout 1
    label fog menu label FOG Project tasking kernel http://192.168.203.113/fog/service/ipxe/arm_Image initrd http://192.168.203.113/fog/service/ipxe/arm_init.cpio.gz append ...
   

   Step Testing Automatic Boot on the Raspberry Pi

    Used the U-Boot sequence with wget and the pxe boot command to attempt dynamic file retrieval at startup.
   
    Technical observation noted: The wget tool embedded in the BusyBox/FOS environment on ARM encounters parsing issues with URLs containing GET parameters (?mac=...), throwing a wget: bad port error.
   

3. Attention Point

   The script generated by boot.php returns a Syslinux/PXE-style configuration format (label fog …), which is natively supported by U-Boot’s pxe boot command. However, full automation via a clean U-Boot command line (bootcmd) requires ensuring that the embedded wget on ARM environments can handle URLs with dynamic parameters without syntax errors.

[Jeremy]

@Tom-Elliott

Thanks for your clarification, and apologies for the confusion. I think there was a misunderstanding about what I am trying to achieve.

I am not trying to bypass FOG or do something outside of FOG — quite the contrary. I am actively using FOG to deploy images to a fleet of Raspberry Pi devices.

My previous confusion came from the fact that my test units kept landing in the interactive FOG debug mode (requiring pressing Enter to get a prompt), which led me down the wrong path thinking I needed a manual workaround. My ultimate goal is simply to have a standard, fully automated deployment task (non-debug) run seamlessly on the Raspberry Pis without requiring manual intervention at each boot.

I will look into standard deployment tasks and post-init scripts as you suggested. Thanks for pointing me in the right direction!

[Tom Elliott]

@Jeremy Two separate things here, and I think they have different answers.

On landing in debug mode: worth being precise about what isdebug actually does, because it’s not what it might look like. The code that runs your postinit script (bin/fog) doesn’t check isdebug at all — it runs whenever fog executes, whether that’s automatic or you typed fog yourself at a shell. What isdebug=yes actually does is two things: it stops FOS from launching fog automatically at boot (you land at the debug info screen and then a bare shell instead), and it turns on every debugPause() — “Press [Enter] to continue” — checkpoint throughout the entire imaging engine, not just one place. There are dozens of them. So even if you manually ran your postinit script from the debug shell and it worked cleanly, anything FOG itself does afterward (a real deploy task) is going to stop and wait for Enter repeatedly, because isdebug=yes turns all of that on at once.

That’s still coming from wherever you’ve got isdebug=yes set — either baked into your boot.cmd/append line, or as a Host Kernel Argument in the FOG web UI for this host, from when you added it to work around the multi-partition issue a few days ago. Clear it in both places. With it gone: fog runs automatically, your postinit script runs (same as before, that part was never conditional), the actual deploy runs with zero pauses, and the box reboots itself when the task completes — no interaction anywhere in the chain. That’s the automation you’re after.

On the wget “bad port” error: worth being precise about where this is happening too — it’s U-Boot itself, at the wget … | pxe boot step, before FOS is even loaded. It isn’t BusyBox and it isn’t anything in FOS or the FOG server; the config I quoted came back correctly from curl, so the server side is fine.

My best read — I don’t have your board to test this, so treat it as a hypothesis to check, not a diagnosis: U-Boot’s wget has two different implementations depending on how it was built. The newer one parses a real URL and splits host from path at the first /, so a query string with colons in it (like your MAC) shouldn’t confuse it. Older builds only understand wget <loadaddr> [<host-ip>:]<path> — no http://, no query-string awareness — and that parser looks for the first colon anywhere in the string, which lands inside mac=dc:a6:… and gets misread as a bogus port.

Can you run wget with no arguments (or help wget) at the U-Boot prompt and paste the usage line it prints? That tells us which of the two you’ve got, and whether this is fixable on your end (newer U-Boot build) or needs the TFTP-staged fallback the docs mention.

[Jeremy]

@Tom-Elliott

I checked both database entries for hostKernelArgs, and isdebug=yes was indeed not present there.

Regarding the U-Boot wget error, I followed your suggestion and went to the U-Boot prompt (=>) on the Raspberry Pi. When I type wget, it responds with:
Unknown command wget try help

So it seems the U-Boot build on these Raspberry Pis does not have the wget command compiled into it at all. What would be the recommended fallback or alternative method to fetch the script from FOG in this case (e.g., using TFTP or an alternative command if available)?"

[Tom Elliott]

@Jeremy Good, that narrows it down — no wget at all in this build, not just the older syntax. Two ways to close that:

Rebuild U-Boot with wget compiled in (keeps you on the dynamic per-host endpoint). CONFIG_CMD_WGET is a plain menuconfig option — you don’t need the HTTPS variant, which pulls in the whole lwIP/mbedTLS stack, just plain HTTP:

make rpi_4_defconfig
make menuconfig # Command line interface -> Network commands -> wget
make

Reflash the resulting u-boot.bin. Since dhcp already works for you, the network stack is there — this just turns on the one command. Once it’s in, the sequence from before should work as documented.

Or skip the dynamic endpoint and hand-roll a static boot.cmd — which is exactly what already worked for your capture: plain tftp for kernel/initrd/dtb, fixed bootargs with the task’s mac=, osid=, type=, img=, etc. baked in. The real tradeoff: tftp doesn’t ask FOG anything, so that config is identical for every board that boots it. Fine if every Pi runs the same task; not fine if you need FOG to decide per-host. boot.php exists specifically to answer “what does this MAC need right now” dynamically, and there’s currently no TFTP equivalent of it for boards that can’t do HTTP — that’s a real gap on FOG’s side, not something you’re doing wrong.

For 100+ boards I’d lean toward rebuilding U-Boot rather than going static, since you keep per-host tasking either way — but that’s your call based on how uniform the fleet’s deployment actually needs to be.

[Jeremy]

@Tom-Elliott "Hi again,

Following your instructions, I checked the U-Boot prompt. When typing wget, it responds with:
Unknown command wget try help
So the U-Boot build on these boards does not include the wget command.

Also, for context on how the board behaves at boot, here is what happens visually during the startup sequence (attached photo): it tries to boot locally, times out on storage (mmc / usb), and then falls back to a standard network BOOTP broadcast loop (Retry time exceeded).

Given that U-Boot lacks wget and relies on standard BOOTP/PXE broadcast rather than direct HTTP fetching, what is the recommended way or fallback mechanism to have these boards pull their configuration from FOG?"