FOG Project Image Capture on Raspberry Pi 4 (ARM64) via U-Boot

[Tom Elliott]

@Jeremy Sorry about the VM. Take the snapshot, and also export the host and image from the web UI (Hosts → Export Host, Images → Export Image), so the next rebuild is an import rather than a retype.

Your screenshot is actually the best news so far, and it answers your “fine-tune” question with “there is nothing to type”:

U-Boot’s DHCP works now. DHCP client bound to address 192.168.203.77 (1 ms) — twice, once inside bootflow scan and once by hand. Whatever was blocking it on the old setup is gone. (The C0A8CB4D.img “File not found” right after is harmless: with no bootfile from DHCP, U-Boot guesses a name from its IP and fails to fetch it. setenv autoload no silences it, but you can’t save that without a card, and it costs nothing.)

bootflow scan is already doing the FOG boot path for you. Look at the top of your screen: after DHCP it asked the TFTP server for pxelinux.cfg/default-arm-bcm283x, then default-arm, then default (and, above what’s visible, 01-88-a2-9e-53-34-c0 first). That is pxe get — U-Boot runs it by itself as part of standard boot, no bootcmd, no environment, no boot.scr. Every one came back “File not found” because FOG only writes the 01-<mac> file while that host has a task queued, and nothing was queued. So the whole automation is: queue the task in the web UI, power the Pi on, done. No typing.

Drop boot.scr entirely. Wrong image format for "source" command is because your boot.scr is plain text; source only runs mkimage-wrapped scripts. You don’t need to fix that, you need to stop using it — the pxe path replaces it.

So, on the rebuilt server, please confirm two things and then do one test:

• FOG Settings → TFTP Server has a FOG_TFTP_ROOT_DIR entry reading /tftpboot. If it doesn’t, the install is older than the fix; run the updater on the beta channel.
• Queue a Deploy task for Raspberry-test, then on the server cat /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 — it should start # Generated by FOG Project and contain kernel arm_Image — and ls -l /tftpboot/arm_Image /tftpboot/arm_init.cpio.gz. Paste both here.
• Then just power-cycle the Pi and let bootflow scan run. It should now find the 01- file, pull arm_Image, and boot into FOS.

On “can’t get an IP via DHCP” during deployment: I need to know which DHCP. U-Boot’s is fine per your screenshot. If you mean FOS itself (after the kernel boots, the FOG banner, Attempting to check in never coming), that’s a different network stack — tell me which kernel booted (FOG’s arm_Image via the 01- file, or kernel8.img from your old script) and paste the lines around udhcpc / eth0. Text rather than a photo if at all possible; a serial adapter on the Pi’s UART pins makes that easy and is worth the few euros for a 200-unit project.

[Jeremy]

@Tom-Elliott

I created a new snapshot to avoid losing data again and having to start everything from scratch; right now, I’m stuck on this page on the Raspberry Pi.

[Tom Elliott]

@Jeremy That screen is the STP pattern, and I’m now fairly confident that’s your DHCP problem. Look at the timing: bootflow scan starts DHCP the instant the link comes up, retries for two cycles (about 20 seconds total), then gives up. Yesterday’s “1 ms” success was typed at the prompt after the board had been sitting there a while. A switch port running classic spanning tree drops everything for roughly 30 seconds after link-up. First 20 seconds: nothing gets through; a minute later: instant lease. That matches every screenshot you’ve posted.

U-Boot can’t be made to wait longer without a saved environment, and without an SD card you can’t save one. So this is fixed on the switch, not on the Pi: ask whoever runs it to enable PortFast (Cisco) / edge port (most others) on the ports the Pis plug into, or move to RSTP. That is a normal setting for any PXE client, not a Pi special case. If you want proof before asking, plug the Pi into a dumb unmanaged switch for one boot; if bootflow scan gets a lease first time, you have your answer.

Two more things from that screen:

• 0 Storage Device(s) found. Last time it found 1 (the USB SSD). If the SSD isn’t attached, there’s nothing to deploy to, so make sure it’s plugged in and powered for the deploy test.
• I still need the three things from my last post, and I can’t go further without them: whether FOG_TFTP_ROOT_DIR exists in FOG Settings → TFTP Server; the contents of /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 after you queue a Deploy task; and ls -l /tftpboot/arm_Image /tftpboot/arm_init.cpio.gz. Those tell me whether the FOG side is doing its job on your rebuilt server. Everything else in this thread is now on the network side.

[Jeremy]

@Tom-Elliott

Regarding OPNsense, there isn’t anyone on the infrastructure team available to look into it right now; I can probably check on that Monday morning.

To be transparent with you, I’ve been retraining for a new career since February 2026 and only just obtained my IT diploma in July. What helps is that I’m persistent with projects and usually manage to get the job done. I’ll get back to you once I have the information regarding OPNsense.

[Tom Elliott]

@Jeremy If you can put a dumb switch between the Pi and the cable you currently connect into:

Connect the cable you normally plugin to the Pi into the 5 port switch then your pi to that switch and this should test the connectivity STP problem I believe is occurring.

[Jeremy]

@Tom-Elliott

Hi, I’m using a Proxmox bridged network interface, I can’t connect anywhere else because otherwise it wouldn’t be the same network, so we’re trying to find a solution on our end.

[Jeremy]

@SimonGuenther

Hi, The blockage isn’t caused by a forgotten debug command, since that isn’t present in the deployment test; we have a managed switch right before the Raspberry Pi connection—that might be the issue. I’m discussing it with my colleagues at work to see how we can handle it.

[Jeremy]

This post is deleted!

[Jeremy]

This post is deleted!

[Jeremy]

@Tom-Elliott

OPNsense:

Option 66: I correctly entered the FOG server’s IP address.

Option 67: I entered the filename ipxe.efi.

Do I always need to keep ipxe.efi in OPNsense?

[Tom Elliott]

@Jeremy
Option 66 -> The FOG server or your TFTP Server.
Option 67 -> The Filename ALL machines should use for booting automatically.

I don’t think this matters in respect to the Raspberry Pi’s though.

[Jeremy]

@Tom-Elliott

Okay, thanks for the reply.

The OPNsense box does assign an IP address to the Raspberry Pi at startup—and it is always the same one, even though it is set to DHCP—but during the U-Boot boot process, it fails to find an IP address.

[Jeremy]

@Tom-Elliott

My Raspeberry is :

RaspberryPI 4 2GB Model B

The settings might vary depending on the model?

[Tom Elliott]

@Jeremy That’s a useful data point, and it says the same thing as before, more firmly: OPNsense hands the Pi’s firmware a lease every time (same address, so you have a static mapping, good), and the only DHCP client that fails is U-Boot’s, which runs seconds after U-Boot resets the network port. The DHCP server is fine. The reservation is fine. What differs is when the request happens relative to the port coming up, and that’s the managed switch your colleagues are now looking at. Nothing in OPNsense will change this.

You can test the whole FOG deploy today, without waiting for anyone. I asked for this last time and haven’t seen the result yet, so once more, because it matters: power the Pi on, let it fail and land at the U-Boot> prompt, wait until it has been powered for a full minute, then type boot. By then the switch port is forwarding, DHCP will bind at once, and standard boot will find your 01- file and pull arm_Image. Whatever appears after Starting kernel ... is the next thing I need to see. This is the same sequence that will run automatically once the switch is fixed; you’re only giving it the head start by hand.

Option 67: the Pi doesn’t use it. The Pi’s bootloader finds the TFTP server from option 66 / next-server and fetches its files by fixed names, and U-Boot’s pxe code only looks at the filename to work out a directory to prefix onto pxelinux.cfg/… — a bare ipxe.efi has no directory, so it prefixes nothing, which is what you want. Two cautions: don’t ever put a path with a folder in there (boot/ipxe.efi would make the Pi look for boot/pxelinux.cfg/01-…), and ipxe.efi as the single filename for all machines is wrong for your x86 fleet — BIOS machines need undionly.kpxe, UEFI ones ipxe.efi, and OPNsense has the per-architecture fields for exactly that. That’s a separate topic; it won’t affect the Pi either way.

One more thing so you don’t chase it: the advice above about mode=debug, running FOS on kernel8.img, VLANs and firewall rules between subnets was written against your old hand-typed setup and doesn’t apply any more. You’re on FOG’s own arm_Image via FOG’s own file now, on one subnet, and the FOG side is already proven on your server. It’s the switch port, then whatever the kernel says after it boots.

[Jeremy]

@Tom-Elliott

I tried your test and am sending you a screenshot; I’ll keep testing on my end, and if I have any good news, I’ll let you know.

[Tom Elliott]

@Jeremy Thanks for running it. The model doesn’t change anything: a Pi 4 Model B is what all of this was written for, and 2 GB is plenty.

From the screenshot I can’t tell whether the minute had passed before you typed boot (it looks like it went straight after the first failure). But rather than argue about seconds, let’s get the one measurement that settles this, and it needs nothing from the infrastructure team: watch the DHCP traffic from the FOG server, which sits on the same network and sees every broadcast.

On the FOG server, as root, start this and leave it running:

tcpdump -ni any -e port 67 or port 68

Then power-cycle the Pi and let U-Boot do its BOOTP broadcast loop. You’ll see one of three things:

• Nothing at all from 88:a2:9e:53:34:c0 while U-Boot is broadcasting → the frames never leave the Pi’s switch port. That’s the switch (STP or port security), full stop. Hand that to your colleagues with the capture.
• Requests arrive but no reply comes back (lines from the Pi’s MAC, none from OPNsense’s) → OPNsense is receiving and ignoring them. Then it’s a DHCP-server question, and the OPNsense DHCP log for that MAC at the same moment will say why.
• Request and reply both appear, but U-Boot still says “Retry time exceeded” → the reply is being dropped on the way back or U-Boot isn’t accepting it. Rare, but it points at the switch again, or at the MAC below.

Copy the tcpdump lines here as text (that’s a terminal, so no photo needed).

One more thing to check at the U-Boot> prompt, since it costs ten seconds:

printenv ethaddr

It must read 88:a2:9e:53:34:c0. U-Boot takes it from the firmware; if for any reason it shows something else, OPNsense’s reservation won’t match, and if your DHCP scope has no dynamic pool that alone would produce exactly this. I don’t expect it, but it’s cheap to rule out.

Once DHCP binds, the rest of the path is already proven on your server, so this really is the last gate before you see FOS boot.

[Jeremy]

This post is deleted!

[Jeremy]

@Tom-Elliott

Here are the two screenshots of the requested tests.

[Tom Elliott]

@Jeremy That capture is the answer we’ve been missing. Thank you — text from a terminal is exactly what I needed.

Read it with me:

• 13:58:46.427 and 13:58:46.457: two packets from 88:a2:9e:53:34:c0, 30 ms apart. That’s the Pi firmware’s DHCP: discover, then request. A clean, complete exchange in under a second (the offer and ack come back unicast, so the FOG server doesn’t see those, which is fine).
• 13:59:53.647: one packet. That’s during U-Boot’s phase. But your screen shows U-Boot broadcasting fifteen or more times across its retry cycles. Only one of them reached the network segment the FOG server is on, and that one wasn’t answered as far as U-Boot could tell.

So: the DHCP server is fine, the reservation is fine, ethaddr is correct (thanks for checking), and FOG’s side is proven. What’s broken is that almost every frame U-Boot sends is disappearing between the Pi and the rest of the LAN, while the firmware’s frames, seconds earlier on the same cable, all get through. That is not a FOG problem and not an OPNsense problem. It’s either the managed switch port treating U-Boot’s traffic differently (broadcast storm control, DHCP snooping rate limits, port security, or STP re-evaluating the port), or U-Boot’s Ethernet driver failing to put most of its packets on the wire. Both are real possibilities, and one test separates them, which is the one I asked for on the 4th and still haven’t seen:

Put a cheap unmanaged switch between the Pi and the wall. Wall cable into the little switch, Pi into the little switch. Power-cycle the Pi with the same tcpdump running on the FOG server.

• If U-Boot’s DHCP now binds and you see its broadcasts arriving in the capture → the managed port is the problem. Give your colleagues the two captures side by side (with and without the little switch) and the port number; that’s everything they need.
• If it still fails and the capture still shows almost nothing from the Pi during U-Boot’s phase → it’s U-Boot’s driver on this board, and we look at which U-Boot build you’re running (version at the prompt, paste the first two lines).

One more piece for Monday, since it’s free: ask whoever runs OPNsense to look at its DHCP log for 88:a2:9e:53:34:c0 around 13:59:53 today. If OPNsense saw that one packet and sent an offer that never made it back, that’s the switch dropping traffic in both directions, and it’s another line of evidence for the port.