FOG Project Image Capture on Raspberry Pi 4 (ARM64) via U-Boot

[Tom Elliott]

@Jeremy Thanks for the screenshots — they changed my read of this quite a bit, and I owe you a correction first.

Correction: earlier I said pxe get “finding” pxelinux.cfg/01-88-a2-9e-53-34-c0 proved the file-generation side worked. It didn’t. U-Boot prints Retrieving file: ... before it sends the request, and the Loading: T T T T that follows means the server never answered at all. A file that really is missing gives you TFTP error: 'File not found', not timeouts. So the file-side was never proven, and it turns out it was broken on my side of the fence.

The FOG bug, now fixed: FOG was writing the 01-<mac> file under the directory it keeps the HTTP-served kernels in (/var/www/html/fog/service/ipxe/), not under /tftpboot. The TFTP daemon runs chrooted to /tftpboot, so the file existed but TFTP could never see it. That’s fixed in working-1.6 (PR #1664) with a new setting, FOG Settings → TFTP Server → FOG_TFTP_ROOT_DIR, which the installer sets to the real TFTP root. Run the updater once it’s merged and check that setting reads /tftpboot; after that, queue a task for the Pi and you should see /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 appear on the server. Please confirm that file is there before the next boot test — it’s the one thing I can check in code but not on your box.

Your current boot loop is a different, earlier step. From the console shot: U-Boot itself loaded cardless, which means the Pi’s EEPROM network boot already pulled the firmware and u-boot.bin from /tftpboot over TFTP. That proves the OPNsense next-server, the FOG TFTP service and the firewall are all fine for this client — so ignore my earlier firewall angle. What’s looping is bootcmd=bootflow scan: that’s U-Boot’s standard-boot sequence, which tries mmc, usb, then ethernet, and the ethernet step starts with U-Boot’s own DHCP request (BOOTP broadcast 1, 2, 3...). That DHCP never succeeds, and it’s never been shown to work in this thread — your earlier manual test set ipaddr/serverip by hand and skipped DHCP entirely. The empty per-architecture filenames in OPNsense are fine, by the way: the pxe boot method doesn’t need a bootfile name at all. (Side note, unrelated to the Pi: the iPXE-class filename in OPNsense is what gets handed to a client that is already running iPXE, and FOG expects default.ipxe there, not ipxe.efi — with ipxe.efi your x86 UEFI clients will reload iPXE forever. Worth a look when you’re back on those.)

Two things to try at the U-Boot prompt, and please paste the exact text rather than a screenshot if you can:

dhcp

on its own. If it also loops on BOOTP, then U-Boot’s network driver isn’t getting a lease and we look at OPNsense’s DHCP log for that MAC (does the request even arrive, does it offer). If it does get an address, then:

pxe get
pxe boot

once the FOG update above is in and the 01-... file is confirmed on disk. If that works by hand, setenv bootcmd 'dhcp; pxe get; pxe boot' and saveenv gets you the automated path — bootflow scan should get there too, but the manual sequence tells us which step is at fault when it doesn’t.

[Jeremy]

@Tom-Elliott

I do have the filename you requested.

When U-Boot starts, DHCP doesn’t work; it spends 8 seconds searching. Sometimes, when rebooting the Raspberry Pi, it manages to obtain an IP address, successfully detects the server’s IP, and downloads pxelinux, but then stops.

What I don’t understand is why U-Boot sometimes manages to get an IP address and sometimes doesn’t.

[Tom Elliott]

@Jeremy Do you happen to have STP on your network?

[Tom Elliott]

@Jeremy Good — the file being there confirms the /tftpboot fix landed. Two separate things in your report, and one of them was mine again.

“Downloads pxelinux, but then stops” — partly expected, partly another FOG bug, now fixed.

• If there was no task queued for the Pi at that moment, the file FOG writes says localboot 0, which tells U-Boot “nothing to do, carry on to local disk”. That’s by design: no task, no imaging. So “stops” after a successful pxe get is correct behavior unless you’d queued a capture first.
• If there was a task queued, it would still have stopped, and that one is on me: the file named the kernel and init as http://… URLs, which is what boards with wget use. Your U-Boot’s pxe code can’t follow a URL — I checked U-Boot’s source (boot/pxe_utils.c every kernel and initrd line is fetched over TFTP, relative to wherever the config came from, so kernel http://… became a TFTP request for a file literally called that. Fixed in working-1.6 (PR #1679): the file now says kernel arm_Image / initrd arm_init.cpio.gz, and FOG copies those two files into /tftpboot itself the first time a task is queued for an ARM host, and again whenever the kernel is updated. Run the updater once more, queue a capture for the Pi, then on the server check that /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 contains kernel arm_Image and that /tftpboot/arm_Image and /tftpboot/arm_init.cpio.gz exist.

The intermittent DHCP is a network-timing problem, not a FOG one, and Tom’s STP question is the right first suspect. On a switch port running classic spanning tree, the port doesn’t forward traffic for roughly 30 seconds after link-up (listening, then learning). U-Boot brings the link up and sends its DHCP discover immediately, retries for a few seconds, and gives up — that’s the “8 seconds searching”. The times it works are the times the port happened to already be forwarding. The Pi’s EEPROM boot succeeds because it retries for much longer. Two fixes, and I’d do both:

• On the switch: enable PortFast / edge-port (or RSTP) on the ports the Pis plug into. That’s the real fix and it helps every PXE client, not just the Pis.
• In U-Boot, so a slow port doesn’t kill the boot anyway:

setenv autoload no
setenv netretry yes
setenv bootcmd 'dhcp; pxe get; pxe boot'
saveenv

autoload no stops dhcp from also trying to TFTP a bootfile it was never given (that’s an extra timeout and a spurious failure in your sequence). netretry yes makes U-Boot keep retrying DHCP instead of giving up after a few seconds — the right call for a headless board that has nothing else to do, but be aware it means a Pi with no DHCP server on the wire will sit there retrying rather than dropping to the prompt.

If DHCP still fails after that, the next read is OPNsense’s DHCP log for that MAC: whether the discover even arrives, and what it offers. And as before, pasted text beats a screenshot — the exact lines around pxe boot are the ones I need next.

[Jeremy]

@Tom-Elliott

Hey, I’m sending you the screenshot of the test.
I’m also sending you the configs to see if I made a mistake.
My colleague told me that STP would be enabled on the switch but not in OPNsense.

Config u-boot

cat /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0

cat /tftpboot/pxelinux.cfg/boot.scr

cat /tftpboot/pxelinux.cfg/default

cat /tftpboot/pxelinux.cfg/default-arm

cat /tftpboot/pxelinux.cfg/default.bak

[Jeremy]

@Tom-Elliott

I’m also showing you screenshots of my host and image to see if I made a mistake.

[Tom Elliott]

@Jeremy Thanks, the file contents are the most useful thing you’ve posted, because they show the 01-… file on your server is not FOG’s. I owe you a correction on that too: yesterday I took “the file is there” as proof the fix landed. It wasn’t — that file is yours.

1. Everything in /tftpboot/pxelinux.cfg/ is hand-written, and it’s the wrong format for pxe boot. 01-88-a2-9e-53-34-c0, boot.scr, default and default-arm all contain a U-Boot script (setenv …, tftp …, booti …). pxe boot doesn’t run scripts; it parses a PXELINUX-style config (label, kernel, initrd, append), finds no labels in yours, and does nothing. That’s your “downloads pxelinux, but then stops”, and it was never going to work regardless of the FOG bugs I fixed. Please don’t hand-write anything under pxelinux.cfg/: FOG owns the 01-<mac> names there, and its periodic reconcile deletes any 01- file whose host has no active task, so your file would vanish anyway. (boot.scr, if you ever wanted one, has to be a mkimage-wrapped binary in the TFTP root, not text under pxelinux.cfg/. You don’t need it; the pxe path is the whole thing.)

To get FOG’s file: run the updater once more (PR #1679 is merged), then in the web UI queue a Capture task for Raspberry-test. Within a moment /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 will be replaced by a file starting # Generated by FOG Project with kernel arm_Image and initrd arm_init.cpio.gz, and /tftpboot/arm_Image and /tftpboot/arm_init.cpio.gz will appear beside it. Paste that file back here. Your host settings are fine: arm64, image set, primary disk /dev/sda. (acpi=off in the kernel arguments does nothing on a Pi; harmless.)

2. saveenv failed because there is no SD card, so nothing you setenv survives a reboot. That’s actually fine for the end goal: the default bootcmd=bootflow scan already does DHCP and then pxe get / pxe boot on its own, no environment needed. So for a cardless fleet you don’t need bootcmd at all. What you do need is DHCP to work inside U-Boot, which brings us to:

3. DHCP. netretry yes was doing its job (“Retry time exceeded; starting again” is it retrying, not failing). With STP confirmed on the switch, the port is most likely blocking for the first ~30 seconds after U-Boot resets the NIC. One test settles it: run setenv netretry yes then dhcp, and wait a full 60 seconds before judging. If it gets a lease on the second or third cycle, that’s STP, and the fix is PortFast / edge-port on the Pi ports (the default bootflow scan won’t retry long enough on its own, and without an SD card you can’t persist netretry). If it still has nothing after 60 seconds, then it isn’t timing: check OPNsense’s DHCP log (Services → DHCPv4 → Log) for 88:a2:9e:53:34:c0 during the attempt and tell me whether a DISCOVER arrives and whether an OFFER goes out.

Once DHCP works and FOG’s file is in place, bootflow scan (or dhcp; pxe get; pxe boot by hand) should pull arm_Image and boot FOS. If it stops anywhere after that, the last lines on screen are what I need — pasted text, not a photo, if there’s any way to capture it.

[Jeremy]

@Tom-Elliott

Hi, I ran into a problem: my VM completely crashed, so I have to start over from scratch. I’m still stuck at the capture stage; Gemini can’t review the entire conversation history we had to get to this point, though I had already reached U-Boot.

When I’ve completed this step, I’ll take a snapshot of the VM this time.

[Jeremy]

@Jeremy

I finally managed to configure the Raspberry Pi image capture; now I just need to fine-tune it to avoid having to enter these commands in the U-Boot terminal:

dhcp
tftp ${scriptaddr} boot.scr
source ${scriptaddr}

Now I’m back to working on the deployment, but I can’t seem to get an IP address via DHCP—I don’t understand why.

[Tom Elliott]

@Jeremy Sorry about the VM. Take the snapshot, and also export the host and image from the web UI (Hosts → Export Host, Images → Export Image), so the next rebuild is an import rather than a retype.

Your screenshot is actually the best news so far, and it answers your “fine-tune” question with “there is nothing to type”:

U-Boot’s DHCP works now. DHCP client bound to address 192.168.203.77 (1 ms) — twice, once inside bootflow scan and once by hand. Whatever was blocking it on the old setup is gone. (The C0A8CB4D.img “File not found” right after is harmless: with no bootfile from DHCP, U-Boot guesses a name from its IP and fails to fetch it. setenv autoload no silences it, but you can’t save that without a card, and it costs nothing.)

bootflow scan is already doing the FOG boot path for you. Look at the top of your screen: after DHCP it asked the TFTP server for pxelinux.cfg/default-arm-bcm283x, then default-arm, then default (and, above what’s visible, 01-88-a2-9e-53-34-c0 first). That is pxe get — U-Boot runs it by itself as part of standard boot, no bootcmd, no environment, no boot.scr. Every one came back “File not found” because FOG only writes the 01-<mac> file while that host has a task queued, and nothing was queued. So the whole automation is: queue the task in the web UI, power the Pi on, done. No typing.

Drop boot.scr entirely. Wrong image format for "source" command is because your boot.scr is plain text; source only runs mkimage-wrapped scripts. You don’t need to fix that, you need to stop using it — the pxe path replaces it.

So, on the rebuilt server, please confirm two things and then do one test:

• FOG Settings → TFTP Server has a FOG_TFTP_ROOT_DIR entry reading /tftpboot. If it doesn’t, the install is older than the fix; run the updater on the beta channel.
• Queue a Deploy task for Raspberry-test, then on the server cat /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 — it should start # Generated by FOG Project and contain kernel arm_Image — and ls -l /tftpboot/arm_Image /tftpboot/arm_init.cpio.gz. Paste both here.
• Then just power-cycle the Pi and let bootflow scan run. It should now find the 01- file, pull arm_Image, and boot into FOS.

On “can’t get an IP via DHCP” during deployment: I need to know which DHCP. U-Boot’s is fine per your screenshot. If you mean FOS itself (after the kernel boots, the FOG banner, Attempting to check in never coming), that’s a different network stack — tell me which kernel booted (FOG’s arm_Image via the 01- file, or kernel8.img from your old script) and paste the lines around udhcpc / eth0. Text rather than a photo if at all possible; a serial adapter on the Pi’s UART pins makes that easy and is worth the few euros for a 200-unit project.

[Jeremy]

@Tom-Elliott

I created a new snapshot to avoid losing data again and having to start everything from scratch; right now, I’m stuck on this page on the Raspberry Pi.

[Tom Elliott]

@Jeremy That screen is the STP pattern, and I’m now fairly confident that’s your DHCP problem. Look at the timing: bootflow scan starts DHCP the instant the link comes up, retries for two cycles (about 20 seconds total), then gives up. Yesterday’s “1 ms” success was typed at the prompt after the board had been sitting there a while. A switch port running classic spanning tree drops everything for roughly 30 seconds after link-up. First 20 seconds: nothing gets through; a minute later: instant lease. That matches every screenshot you’ve posted.

U-Boot can’t be made to wait longer without a saved environment, and without an SD card you can’t save one. So this is fixed on the switch, not on the Pi: ask whoever runs it to enable PortFast (Cisco) / edge port (most others) on the ports the Pis plug into, or move to RSTP. That is a normal setting for any PXE client, not a Pi special case. If you want proof before asking, plug the Pi into a dumb unmanaged switch for one boot; if bootflow scan gets a lease first time, you have your answer.

Two more things from that screen:

• 0 Storage Device(s) found. Last time it found 1 (the USB SSD). If the SSD isn’t attached, there’s nothing to deploy to, so make sure it’s plugged in and powered for the deploy test.
• I still need the three things from my last post, and I can’t go further without them: whether FOG_TFTP_ROOT_DIR exists in FOG Settings → TFTP Server; the contents of /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 after you queue a Deploy task; and ls -l /tftpboot/arm_Image /tftpboot/arm_init.cpio.gz. Those tell me whether the FOG side is doing its job on your rebuilt server. Everything else in this thread is now on the network side.

[Jeremy]

@Tom-Elliott

Regarding OPNsense, there isn’t anyone on the infrastructure team available to look into it right now; I can probably check on that Monday morning.

To be transparent with you, I’ve been retraining for a new career since February 2026 and only just obtained my IT diploma in July. What helps is that I’m persistent with projects and usually manage to get the job done. I’ll get back to you once I have the information regarding OPNsense.

[SimonGuenther]

@Jeremy Haven’t done this on a Pi specifically, so take this as pointers rather than a known fix, but a few things jump out.

First, if udhcpc is already getting a lease, then the FOS init is running, so it’s not really a kernel hang, it’s stopping somewhere in the init scripts after network up. Which makes me look straight at mode=debug in your bootargs. Debug mode in FOS is interactive, it waits for you to hit a key or type at a shell before it does anything. If the console you’re watching isn’t the one it’s prompting on, that looks exactly like a hang. I’d drop mode=debug for a run and see if it moves on, and separately get a serial console on the Pi (enable_uart in config.txt, add console=serial0,115200 alongside your tty1) so you can actually see where it stops instead of guessing from HDMI.

Second, you swapped the kernel but kept the stock FOS initrd. That’s fine in principle, but the FOG init was built against FOG’s own kernel config. Worth checking that the Pi kernel has everything the init expects when it mounts root from /dev/ram0 with that ramdisk_size, and that nothing later in the init is relying on a module or kernel feature the Pi build doesn’t ship. A serial log with loglevel=7 will tell you quickly.

Third, unrelated to the hang but it’ll bite you right after: for a capture (type=up) FOG normally points storage at the dev share, storage=IP:/images/dev/, not /images/. The image gets moved into /images/ by the server once the upload finishes. If you hand rolled the bootargs from a deploy example, that one’s easy to miss.

Also, since the target is a single USB SSD, you might get more mileage checking whether the FOS init even handles /dev/sda as a capture target the way you’d expect on ARM, but that’s step two. Get it past checkin first. Would be good to hear if Tom or Sebastian have opinions on running FOS with a non FOG kernel, I don’t think that’s a path many have gone down.

[Tom Elliott]

@Jeremy If you can put a dumb switch between the Pi and the cable you currently connect into:

Connect the cable you normally plugin to the Pi into the 5 port switch then your pi to that switch and this should test the connectivity STP problem I believe is occurring.