FOG Project Image Capture on Raspberry Pi 4 (ARM64) via U-Boot

[Jeremy]

@Jeremy said in FOG Project Image Capture on Raspberry Pi 4 (ARM64) via U-Boot:

@Tom-Elliott

GREAT NEWS! The capture completed successfully!

Here is a summary of the steps and final configuration that made it work seamlessly on the Raspberry Pi 4:

1. Display Fix (Kernel side):
   Downloading build EXP_20260827-114033 with the integrated VC4 driver completely resolved the display freeze on the 4 raspberries logo. HDMI output initialized perfectly.

2. Ramdisk Format Fix (U-Boot side):
   Passing raw arm_init.cpio.gz via booti was triggering a “Wrong Ramdisk Image Format” error in U-Boot. Wrapping it into uInitrd with the following command solved the issue:
   mkimage -A arm64 -O linux -T ramdisk -C none -d arm_init.cpio.gz uInitrd

3. Final Working boot.cmd:

---

setenv kernel_addr_r 0x00080000
setenv fdt_addr_r 0x08000000
setenv ramdisk_addr_r 0x09000000

setenv bootargs console=tty1 loglevel=7 consoleblank=0 web=http://192.168.203.113/fog/ mac=dc:a6:32:c8:b4:33 osid=50 type=up img=ImagePi4 imgType=mps imgPartitionType=all storage=192.168.203.113:/images/dev/ storageip=192.168.203.113 fdrive=/dev/sda

tftp ${kernel_addr_r} 7efcf632/arm_Image
tftp ${ramdisk_addr_r} 7efcf632/uInitrd
tftp ${fdt_addr_r} 7efcf632/bcm2711-rpi-4-b.dtb

booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}

4. Imaging Status:
   Once booted into FOS, Partclone captured /dev/sda without any error and uploaded the full image to the FOG server!

Thank you so much for adding the VC4 driver and supporting ARM64 builds!

[Tom Elliott]

@Jeremy That’s the first time anyone has captured an image from a Raspberry Pi
with FOG, so thank you for sticking with it through three rounds of this. The
vc4 change is merged and will be in the next build.

One correction to your boot.cmd before someone copies it: you didn’t actually
need the uInitrd wrapper. The “Wrong Ramdisk Image Format” error came from
dropping the size off the ramdisk argument - without it U-Boot reads the address
as a legacy uImage and insists on a header. Either of these works:

booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

or the wrapper you built, which is fine as-is because you used -C none. That
part matters: -C gzip would have U-Boot decompress it and the kernel’s own gzip
support - the first bug we fixed in this thread - would never have been
exercised.

For anyone finding this later, the display problem was ours. We build without
DRM everywhere and rely on the firmware handing us a framebuffer, which is safe
on a PC because UEFI always publishes one. On a Pi it isn’t: our device tree has
no simple-framebuffer node, and the one the firmware would normally inject
disappears as soon as config.txt enables vc4-kms-v3d, which current Pi OS ships
turned on. So the kernel was booting fine the whole time and had no way to tell
anyone.

Worth saying plainly: your report exercised three separate fixes on real
hardware for the first time - the initramfs decompression, the ARM platform
support, and the NFS mount options. All of it had only ever run in QEMU before
you.

[Jeremy]

@Tom-Elliott

Thank you so much for the detailed explanation and for all your hard work on this!

It’s amazing to know that this was a first for the Raspberry Pi and FOG Project, and I’m really glad my tests helped validate those three major fixes on real hardware.

Good catch on the :filesize argument syntax for booti as well. That clarification will definitely be super helpful for anyone setting up ARM64/Pi deployments in the future!

Everything is working smoothly on my end now. Thanks again for the incredible reactivity and support.

The next test is to deploy the image to other Raspberry Pis; I’ll keep you posted.

[Jeremy]

@Tom-Elliott

I’ve been working on deploying images to a Raspberry Pi 4 (ARM64) using FOG (version 1.5.10.2254, FOS kernel 6.18.38), and I wanted to share our findings, encountered errors, and current limitations we faced during the process. Hopefully, this can help improve native ARM support or guide others trying to achieve unattended RPi deployments.

Here is a summary of what we experienced and how we bypassed it:

1. U-Boot bootargs Buffer Overflow

   Issue: When adding extra environment variables to bootargs inside boot.cmd (such as custom storage IPs, disk targets, or MAC addresses), the ARM64 kernel command line exceeded the maximum allowed size. This caused the kernel boot process to silently halt right after the network interface came up (Link is Up).

   Workaround: Kept the bootargs string as minimal as possible and relied on FOG web settings where applicable.

2. Missing osid Parameter

   Issue: FOS initialization stops with No os id passed (determineOS) and reboots if the kernel parameters do not explicitly provide osid=50 (or the corresponding OS ID) alongside type=down.

   Workaround: Explicitly appending osid=50 type=down in the U-Boot script parameters.

3. FOG Automatic Script Orchestration vs. ARM Multi-Partition Layout

   Issue: Even when successfully booting into FOS and passing the correct variables (img=Raspberry, storage=…, fdrive=/dev/sda), the automated fog script skips or fails to correctly parse multi-partition layouts on USB/SD external drives (/dev/sda), directly jumping to “Task Complete” without actually writing data via Partclone or triggering errors like Fatal Error: Unknown request type :: Null.

   Workaround: We had to enter FOG Debug Mode (isdebug=yes), export variables manually (export storage, export img, export type=down), and notice that FOG’s single-partition versus multi-partition routines expect specific raw file structures (d1.mbr, d1p1.img, d1p2.img).

4. Raw Image Files (.img) vs. Partclone Format

   Issue: When attempting manual deployment via partclone.restore, Partclone throws This is not partclone image because FOG stores standard multi-partition RPi captures as raw binary dumps (.img) rather than compressed partclone streams.

   Workaround: Restoring the MBR and partitions required direct block-level commands:
   Bash

   dd if=/images/Raspberry/d1.mbr of=/dev/sda bs=512 count=1
   dd if=/images/Raspberry/d1p1.img of=/dev/sda1 bs=4M status=progress
   dd if=/images/Raspberry/d1p2.img of=/dev/sda2 bs=4M status=progress

   Result: While the partitions are written, the RPi firmware still struggles to directly boot the resulting USB structure (Unable to read partition as FAT), indicating that standard FOG MBR generation (d1.mbr) doesn’t align cleanly with RPi’s expected GPT/FAT bootloader requirements for USB-MSD booting.

It seems the automated deployment engine (fog script) inside FOS needs specific adaptations for ARM/Raspberry Pi multi-partition disk imaging (handling fdrive=/dev/sda and raw dd-like partition layouts seamlessly).

Any insights or recommendations on how to properly handle native RPi deployments through FOG would be greatly appreciated!

[Tom Elliott]

@Jeremy Stop the dd approach - it’s what’s breaking the disk, not the Pi.

Those .img files are not raw dumps. They’re zstd-compressed partclone streams;
check any of them and you’ll see the zstd magic 28 b5 2f fd in the first four
bytes. So dd’ing d1p1.img onto /dev/sda1 writes compressed bytes straight onto
the partition, which is exactly why the firmware then can’t read it as FAT.
That’s also why partclone.restore told you it wasn’t a partclone image - it was
looking at the zstd wrapper. FOS does this at funcs.sh:847:

zstdmt -dc /images/Raspberry/d1p1.img | partclone.restore --ignore_crc -O /dev/sda1 -N -f 1

One per partition. If the image was captured with a gzip format instead, swap
zstdmt for pigz -dc.

The MBR is wrong too. d1.mbr isn’t one sector - we capture everything up to the
first partition, capped at 1MiB, so it’s usually 2048 sectors. Your count=1
threw away everything between the boot sector and the first partition. Drop the
count entirely and let it write the whole file:

dd if=/images/Raspberry/d1.mbr of=/dev/sda bs=512

Now the more useful finding. “Fatal Error: Unknown request type :: Null” is
FOS telling you $type arrived empty, and that’s the same problem as your
missing osid - your bootargs are being truncated before FOS ever reads them. It
isn’t the kernel: arm64’s COMMAND_LINE_SIZE is 2048, the same as x86, and the
bootargs line you posted is 233 characters. Look at U-Boot’s own command and
console buffer instead, that’s where the ceiling is on your setup.

Which gets to the real limitation, and I’d rather be straight about it: FOG’s
server has no U-Boot support at all. boot.php emits iPXE, and nothing generates
a boot.scr, delivers a DTB, or builds arguments for a non-iPXE client. Every
variable you’re hand-assembling is one the server would normally hand the
client, and doing it by hand is why the ordering and length problems are yours
to fight.

Two ways forward. Keep hand-rolling U-Boot, in which case use the two commands
above and keep bootargs minimal. Or put UEFI firmware on the Pi (pftf/RPi4) so
it PXE-boots into iPXE like any other machine and the server drives the whole
thing - that’s the route that already works today, and it’s where native
support would build from.

Capture works, which is the half that was broken. Deploy needs the server side
to exist.

[Tom Elliott]

@Jeremy Last time I told you FOG’s server had no U-Boot support and your two
options were to keep hand-rolling it or put UEFI firmware on the Pi. That’s no
longer true - I built the server side. It’s merged into working-1.6 as of
1.6.0-beta.4318, so update and you’ll have it.

There’s a new endpoint:

http://<fogserver>/fog/service/uboot/boot.php?mac=<the Pi's MAC>

It answers with an extlinux.conf-style document, computed live from the
database. If the host has a task queued you get kernel/initrd/append for FOS;
if it doesn’t, you get localboot and U-Boot falls through to the disk. Fetch it
with curl first - the whole thing is plain text and readable, which is the
point.

On the Pi:

dhcp
setenv pxefile_addr_r 0x02000000
wget ${pxefile_addr_r} http://<fogserver>/fog/service/uboot/boot.php?mac=${ethaddr}
pxe boot ${pxefile_addr_r}

Use pxe boot, not sysboot. Every extlinux example you’ll find online shows
sysboot, and it’s wrong here - sysboot reads a config off a filesystem on a
block device, so it can’t see what wget just put in memory and does nothing
visible. That’s the one I’d expect to catch people out.

This should also fix your truncation. “Unknown request type :: Null” was $type
arriving empty because your bootargs were being cut short, and I pointed at
U-Boot’s command buffer rather than the kernel. pxe boot doesn’t go through
that buffer - it parses the append line out of the loaded file straight into
bootargs, so the ceiling you were hitting isn’t in the path any more. Worth
confirming, since I’m reasoning about your build rather than looking at it.

Two deliberate omissions, both because they’re properties of your board and not
of FOG:

FOG serves no device tree. No fdt or fdtdir directive, so you keep the tree at
${fdtcontroladdr} - the one VideoCore just built for the exact board revision
it’s running on, which is better than anything I could serve you. Load
addresses stay yours for the same reason.

And FOG writes no pxelinux.cfg files. The endpoint is stateless: the board asks
by MAC and gets an answer out of the database, so there’s nothing on disk to
drift from what’s actually queued and nothing to clean up when a task finishes.

One thing to check before you start: if your FOG install has netboot set to
HTTPS, none of this will work. U-Boot’s wget is HTTP-only with no TLS at all,
so it can’t even fail a certificate check - the redirect just ends the boot with
nothing on screen. The installer now exempts service/uboot/ from the HTTP-to-
HTTPS redirect the same way it does service/ipxe/, but only when netboot is
configured for HTTP in the first place.

Full write-up is in docs/UBOOT_ARM_BOOT.md in the repo.

Being straight about what this is: the emitted config is pinned byte-for-byte
by tests and the decisions behind it are the same code the iPXE path runs, so I
know FOG emits what I intended. What I can’t tell you is that a real U-Boot
consumes it, because nobody here has a Pi. You’re the only person who can prove
that, and if it’s wrong I’d rather hear it than not.

[Jeremy]

This post is deleted!

[Tom Elliott]

@Jeremy you seem to be trying to use fogs boot system to do your own thing. What you’re asking is perfectly possible just not via debug.

Debug has a very specific purpose so to be trying to say: “hey can you stop doing what your program is intended to do so I can do things I want using your program?” Isn’t a good use of anyone’s time, particularly the developers.

What you want to do is already possible, but you’ll have to make your own post init script.

Seeing as what you want to do is not have a fog server, but to do something else, I cannot tell you what needs to happen.

You will need to read the documentation, lax as it may be, create a custom task (not debug) to do your work autonomously, and learn how the post init scripts work.

[Jeremy]

@Tom-Elliott

1. Context and Objective

   Hardware: Raspberry Pi (ARM architecture) running U-Boot.

   FOG Server: Updated to the working-1.6 branch (tested version: 1.6.0-beta.4707).

   Objective: Enable fully automated and dynamic deployment/boot (without human intervention or blocking debug mode) using the new U-Boot endpoint created by the developer (/fog/service/uboot/boot.php).

2. Testing Process and Results

   Step A: Initial Manual Validation (Success)

    By executing the deployment commands manually on the Raspberry Pi terminal, the image cloning completed successfully, and the image was properly copied to the new Raspberry Pi. This confirmed that the hardware and image transfer parts are functional.
   

   Step B: FOG Server Update

    Updated the server to the working-1.6 branch and executed ./bin/installfog.sh to integrate the new U-Boot endpoint.
   

   Step Testing the U-Boot Endpoint (boot.php)

    The curl request executed on the FOG server for a specific MAC address:
    Bash
   
    curl "http://127.0.0.1/fog/service/uboot/boot.php?mac=88:a2:9e:53:34:c0"
   
    Server Response: The endpoint correctly generates and returns a script in PXE/Syslinux format structured as follows:
    Plaintext
   
    # Generated by FOG Project. Do not edit -- it is not stored.
    default fog
    timeout 1
    label fog menu label FOG Project tasking kernel http://192.168.203.113/fog/service/ipxe/arm_Image initrd http://192.168.203.113/fog/service/ipxe/arm_init.cpio.gz append ...
   

   Step Testing Automatic Boot on the Raspberry Pi

    Used the U-Boot sequence with wget and the pxe boot command to attempt dynamic file retrieval at startup.
   
    Technical observation noted: The wget tool embedded in the BusyBox/FOS environment on ARM encounters parsing issues with URLs containing GET parameters (?mac=...), throwing a wget: bad port error.
   

3. Attention Point

   The script generated by boot.php returns a Syslinux/PXE-style configuration format (label fog …), which is natively supported by U-Boot’s pxe boot command. However, full automation via a clean U-Boot command line (bootcmd) requires ensuring that the embedded wget on ARM environments can handle URLs with dynamic parameters without syntax errors.

[Jeremy]

@Tom-Elliott

Thanks for your clarification, and apologies for the confusion. I think there was a misunderstanding about what I am trying to achieve.

I am not trying to bypass FOG or do something outside of FOG — quite the contrary. I am actively using FOG to deploy images to a fleet of Raspberry Pi devices.

My previous confusion came from the fact that my test units kept landing in the interactive FOG debug mode (requiring pressing Enter to get a prompt), which led me down the wrong path thinking I needed a manual workaround. My ultimate goal is simply to have a standard, fully automated deployment task (non-debug) run seamlessly on the Raspberry Pis without requiring manual intervention at each boot.

I will look into standard deployment tasks and post-init scripts as you suggested. Thanks for pointing me in the right direction!

[Tom Elliott]

@Jeremy Two separate things here, and I think they have different answers.

On landing in debug mode: worth being precise about what isdebug actually does, because it’s not what it might look like. The code that runs your postinit script (bin/fog) doesn’t check isdebug at all — it runs whenever fog executes, whether that’s automatic or you typed fog yourself at a shell. What isdebug=yes actually does is two things: it stops FOS from launching fog automatically at boot (you land at the debug info screen and then a bare shell instead), and it turns on every debugPause() — “Press [Enter] to continue” — checkpoint throughout the entire imaging engine, not just one place. There are dozens of them. So even if you manually ran your postinit script from the debug shell and it worked cleanly, anything FOG itself does afterward (a real deploy task) is going to stop and wait for Enter repeatedly, because isdebug=yes turns all of that on at once.

That’s still coming from wherever you’ve got isdebug=yes set — either baked into your boot.cmd/append line, or as a Host Kernel Argument in the FOG web UI for this host, from when you added it to work around the multi-partition issue a few days ago. Clear it in both places. With it gone: fog runs automatically, your postinit script runs (same as before, that part was never conditional), the actual deploy runs with zero pauses, and the box reboots itself when the task completes — no interaction anywhere in the chain. That’s the automation you’re after.

On the wget “bad port” error: worth being precise about where this is happening too — it’s U-Boot itself, at the wget … | pxe boot step, before FOS is even loaded. It isn’t BusyBox and it isn’t anything in FOS or the FOG server; the config I quoted came back correctly from curl, so the server side is fine.

My best read — I don’t have your board to test this, so treat it as a hypothesis to check, not a diagnosis: U-Boot’s wget has two different implementations depending on how it was built. The newer one parses a real URL and splits host from path at the first /, so a query string with colons in it (like your MAC) shouldn’t confuse it. Older builds only understand wget <loadaddr> [<host-ip>:]<path> — no http://, no query-string awareness — and that parser looks for the first colon anywhere in the string, which lands inside mac=dc:a6:… and gets misread as a bogus port.

Can you run wget with no arguments (or help wget) at the U-Boot prompt and paste the usage line it prints? That tells us which of the two you’ve got, and whether this is fixable on your end (newer U-Boot build) or needs the TFTP-staged fallback the docs mention.

[Jeremy]

@Tom-Elliott

I checked both database entries for hostKernelArgs, and isdebug=yes was indeed not present there.

Regarding the U-Boot wget error, I followed your suggestion and went to the U-Boot prompt (=>) on the Raspberry Pi. When I type wget, it responds with:
Unknown command wget try help

So it seems the U-Boot build on these Raspberry Pis does not have the wget command compiled into it at all. What would be the recommended fallback or alternative method to fetch the script from FOG in this case (e.g., using TFTP or an alternative command if available)?"

[Tom Elliott]

@Jeremy Good, that narrows it down — no wget at all in this build, not just the older syntax. Two ways to close that:

Rebuild U-Boot with wget compiled in (keeps you on the dynamic per-host endpoint). CONFIG_CMD_WGET is a plain menuconfig option — you don’t need the HTTPS variant, which pulls in the whole lwIP/mbedTLS stack, just plain HTTP:

make rpi_4_defconfig
make menuconfig # Command line interface -> Network commands -> wget
make

Reflash the resulting u-boot.bin. Since dhcp already works for you, the network stack is there — this just turns on the one command. Once it’s in, the sequence from before should work as documented.

Or skip the dynamic endpoint and hand-roll a static boot.cmd — which is exactly what already worked for your capture: plain tftp for kernel/initrd/dtb, fixed bootargs with the task’s mac=, osid=, type=, img=, etc. baked in. The real tradeoff: tftp doesn’t ask FOG anything, so that config is identical for every board that boots it. Fine if every Pi runs the same task; not fine if you need FOG to decide per-host. boot.php exists specifically to answer “what does this MAC need right now” dynamically, and there’s currently no TFTP equivalent of it for boards that can’t do HTTP — that’s a real gap on FOG’s side, not something you’re doing wrong.

For 100+ boards I’d lean toward rebuilding U-Boot rather than going static, since you keep per-host tasking either way — but that’s your call based on how uniform the fleet’s deployment actually needs to be.

[Jeremy]

@Tom-Elliott "Hi again,

Following your instructions, I checked the U-Boot prompt. When typing wget, it responds with:
Unknown command wget try help
So the U-Boot build on these boards does not include the wget command.

Also, for context on how the board behaves at boot, here is what happens visually during the startup sequence (attached photo): it tries to boot locally, times out on storage (mmc / usb), and then falls back to a standard network BOOTP broadcast loop (Retry time exceeded).

Given that U-Boot lacks wget and relies on standard BOOTP/PXE broadcast rather than direct HTTP fetching, what is the recommended way or fallback mechanism to have these boards pull their configuration from FOG?"

[Tom Elliott]

@Jeremy Good news on the wget front — I closed that gap I mentioned. FOG can
now serve boards with no wget at all, over plain TFTP instead of HTTP.

U-Boot’s pxe get command doesn’t need wget — every U-Boot build has it,
and unlike wget it isn’t configurable: it always fetches
pxelinux.cfg/01-<mac, dash-separated, lowercase> by TFTP from whatever
server your board’s serverip points at. FOG now keeps a real file at that
path for every host, kept in sync automatically whenever a task is queued,
completed, or canceled — so your boot sequence becomes just:

dhcp
pxe get
pxe boot

No URL, no query string, nothing for the “bad port”/no-wget class of problem
to trip over — the filename is fixed by the MAC, so there’s nothing to
substitute.

This just merged into working-1.6 — same branch you’re already tracking
(you quoted 1.6.0-beta.4707 a few hours ago; this landed after that), so
running your update again should pick it up. Two things to check once you
have:

• Settings → FTP/TFTP needs a TFTP host/username/password configured — same
  connection FOG already uses to upload your ARM kernel, so if that’s
  already working for you, you’re set.
• I still don’t have a Pi to test this against, same as everything else in
  this thread — if pxe get doesn’t find the file, or finds one FOG didn’t
  mean to serve, that’s exactly the kind of thing I need you to report back.

One thing from your last post I want to flag separately, because it looks
like a different problem from the wget one: the boot sequence you
described — times out on local storage, then falls into a BOOTP broadcast
loop that gives up with “Retry time exceeded” — reads like U-Boot’s own
default bootcmd running, not a custom one. That’s the standard
Raspberry Pi fallback behavior when nothing else answers, and it wouldn’t
run dhcp / pxe get / pxe boot at all, custom or not.

Everything you’ve described testing so far — the wget/pxe boot sequence,
the curl check, now pxe get — sounds like it’s been typed by hand at the
=> prompt each time. That proves the commands work, but it can’t be “fully
automated,” which was the goal from your very first post: nothing runs
automatically unless it’s persisted as bootcmd, either with
setenv bootcmd '...'; saveenv on the board itself, or as a boot.scr /
extlinux.conf your SD card or TFTP server hands U-Boot on its own at power-on.

Can you confirm which of those two you actually have in place right now? If
the answer is “neither yet — I’ve only been testing at the prompt,” that’s
probably the real blocker to automation, independent of wget vs. TFTP, and
worth sorting out before we look any further at the fetch mechanism itself.

[Jeremy]

@Tom-Elliott

"To answer your question directly: yes, so far I have only been testing manually at the U-Boot prompt (=>).

My main concern right now is scalability. Since I need to deploy this across a fleet of 100 to 200 Raspberry Pi units, configuring each SD card manually or typing commands at the prompt for every single machine is not an option.

What is the recommended best practice in FOG working-1.6 to handle automation for a large fleet? Can the bootcmd be permanently stored in the Pi’s onboard EEPROM once, or can U-Boot fetch a boot script via TFTP automatically without manual card preparation?"

[Jeremy]

@Tom-Elliott

"Hi again! Great news regarding the new TFTP fallback mechanism.

I tested it manually by setting the IP addresses:
Plaintext

setenv ipaddr 192.168.203.50
setenv serverip 192.168.203.113
pxe get

What happened:

U-Boot successfully found the correct path generated by FOG based on the MAC address (Retrieving file: pxelinux.cfg/01-88-a2-9e-53-34-c0).

However, the download times out (Loading: T T T T T T) and results in a Retry count exceeded.

For context: a deployment task was explicitly active and queued in the FOG web interface before running the test, so the file should be there.

Is there a specific TFTP block size or setting required in working-1.6 for U-Boot’s pxe get to successfully pull the file without timing out, or could it be related to firewall/TFTP service configuration on the server side?"

[Tom Elliott]

@Jeremy That’s real progress — pxe get finding pxelinux.cfg/01-88-a2-9e-53-34-c0 by MAC means the naming convention and the file-generation side both work end to end. That’s the part I couldn’t test myself, so thank you for actually running it.

On the timeout itself: “found the file, then hung mid-transfer” (Retrieving file: ... succeeding, then Loading: T T T T T T / Retry count exceeded) points at TFTP’s own connection model, not at FOG. TFTP’s initial request goes to port 69, but the server then hands the rest of that transfer off to a different, randomly chosen UDP port for the actual data — so if there’s a firewall between the Pi and your FOG server that only allows port 69 through, the request succeeds (that’s why U-Boot found the file) and every packet after it gets silently dropped. That matches your symptom closely enough that I’d check it first: either open the OS’s ephemeral UDP range to your TFTP daemon, or (simpler if this is on the same box) confirm nothing’s filtering loopback/LAN traffic to it at all. If you’ve got shell on the FOG server, tcpdump -ni <iface> port 69 or portrange 32768-60999 while you retry pxe get should show the request going out and then nothing coming back, which would confirm it.

Separately — while looking into this I found a real gap on FOG’s side and fixed it regardless of whether it’s your actual cause: the file pxe get downloads wasn’t having its permissions set after upload, so if your TFTP daemon reads as a different user than the account FOG uploads over SFTP with, that alone could produce exactly this symptom (found the file, denied reading it). That fix is in the same working-1.6 branch now, so another update will pick it up. Worth ruling both things out — they’re not mutually exclusive.

On your scalability question: for 100–200 units, don’t script the SD card at all — the RPi 4’s own boot EEPROM supports network boot as the first boot mode, ahead of SD/USB, which is exactly what you want instead of the local-storage-timeout-then-BOOTP-fallback sequence you saw. rpi-eeprom-config (or the raspi-config boot order menu) lets you set that once, and once it’s set it’s a firmware-level property of the board, not something FOG or an SD card image controls — one flash per unit, and after that every power-on goes straight to network boot, no bootcmd persistence step needed on your end at all. I’d flash that as part of whatever imaging/provisioning process gets each Pi ready before it ever talks to FOG the first time, rather than trying to solve it per-deployment.

I don’t have a Pi 4 here to confirm the exact EEPROM boot-order syntax for your firmware revision, so double-check against Raspberry Pi’s own EEPROM config docs before you commit to a fleet-wide flash — but the “make network boot the default at the firmware level” shape of the answer is what actually gets you out of typing anything at the prompt.

[Tom Elliott]

@Jeremy Correction on the firewall angle: if your server went through FOG’s own installer on ufw or firewalld, that’s actually less likely to be it than I made it sound — the installer already handles the exact “TFTP works for the request but the reply is on a random port” problem for both of those (a named tftp service on firewalld, and nf_conntrack_tftp loaded for ufw), specifically because it’s bitten people before.

So rather than “check your firewall” in general, four narrower things:

1. Which firewall are you actually running — ufw, firewalld, or plain iptables? The installer only auto-configures the first two; if it’s bare iptables, my original theory stands and you’d need to allow the ephemeral UDP range by hand.
2. If it’s ufw: lsmod | grep tftp — the installer tries to load nf_conntrack_tftp but silently continues if that fails, so it’s worth confirming it’s actually loaded rather than assumed.
3. Is BOOT_external_tftp_server set to “yes” in your FOG settings? If so and your TFTP is actually running on the FOG box itself, firewalld would have skipped opening it for you.
4. Are the Pi and the FOG server on the same subnet/VLAN, or is there a router, switch ACL, or (if this is a VM/cloud box) a security group between them? None of the above touches anything outside this box.

If you’ve got shell on the FOG server, tcpdump -ni <iface> port 69 or portrange 32768-60999 while you retry pxe get is still the fastest way to see whether the request goes out and nothing comes back (firewall/network) versus the request itself failing (something else entirely).

[Jeremy]

@Tom-Elliott

Here is an update following our latest troubleshooting sessions and tests:

The !V@JL artifact: That part was indeed a leftover artifact from editing the file manually with nano. Once cleaned up, we hit a regression: the Raspberry Pi no longer downloads the 563-byte config file at all anymore. It now loops directly on BOOTP broadcast and times out without hitting the TFTP stage (Retrieving file... doesn't trigger anymore).

Goal & Architecture: The target deployment must remain 100% cardless (no local storage, no SD card, no USB drive used during boot), relying purely on the RPi 4 network boot EEPROM + OPNsense DHCP + FOG/U-Boot.

Regarding your earlier notes on TFTP/Firewall:

    We are running UFW on the FOG server, and we verified that nf_conntrack_tftp is active (lsmod | grep tftp confirmed).

    BOOT_external_tftp_server is not enabled (TFTP runs locally on the FOG box).

    The Pi and FOG server are on the same subnet, with no intermediate restrictive ACLs on that local segment.

The permission fix: We noted your mention about the file permissions fix for downloaded files not being set properly after upload in the working-1.6 branch. Could this regression (dropping back to infinite BOOTP loops instead of attempting the TFTP fetch) be tied to how the file is currently served or indexed, or is there a specific way U-Boot expects the handoff after DHCP?