• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    FOG Project Image Capture on Raspberry Pi 4 (ARM64) via U-Boot

    Scheduled Pinned Locked Moved Unsolved FOG Problems
    47 Posts 3 Posters 978 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      Jeremy @Tom Elliott
      last edited by

      This post is deleted!
      Tom ElliottT 1 Reply Last reply Reply Quote 0
      • Tom ElliottT
        Tom Elliott @Jeremy
        last edited by

        @Jeremy you seem to be trying to use fogs boot system to do your own thing. What you’re asking is perfectly possible just not via debug.

        Debug has a very specific purpose so to be trying to say: “hey can you stop doing what your program is intended to do so I can do things I want using your program?” Isn’t a good use of anyone’s time, particularly the developers.

        What you want to do is already possible, but you’ll have to make your own post init script.

        Seeing as what you want to do is not have a fog server, but to do something else, I cannot tell you what needs to happen.

        You will need to read the documentation, lax as it may be, create a custom task (not debug) to do your work autonomously, and learn how the post init scripts work.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        J 1 Reply Last reply Reply Quote 0
        • J
          Jeremy @Tom Elliott
          last edited by

          @Tom-Elliott

          1. Context and Objective

            Hardware: Raspberry Pi (ARM architecture) running U-Boot.

            FOG Server: Updated to the working-1.6 branch (tested version: 1.6.0-beta.4707).

            Objective: Enable fully automated and dynamic deployment/boot (without human intervention or blocking debug mode) using the new U-Boot endpoint created by the developer (/fog/service/uboot/boot.php).

          2. Testing Process and Results

            Step A: Initial Manual Validation (Success)

             By executing the deployment commands manually on the Raspberry Pi terminal, the image cloning completed successfully, and the image was properly copied to the new Raspberry Pi. This confirmed that the hardware and image transfer parts are functional.
            

            Step B: FOG Server Update

             Updated the server to the working-1.6 branch and executed ./bin/installfog.sh to integrate the new U-Boot endpoint.
            

            Step 😄 Testing the U-Boot Endpoint (boot.php)

             The curl request executed on the FOG server for a specific MAC address:
             Bash
            
             curl "http://127.0.0.1/fog/service/uboot/boot.php?mac=88:a2:9e:53:34:c0"
            
             Server Response: The endpoint correctly generates and returns a script in PXE/Syslinux format structured as follows:
             Plaintext
            
             # Generated by FOG Project. Do not edit -- it is not stored.
             default fog
             timeout 1
             label fog menu label FOG Project tasking kernel http://192.168.203.113/fog/service/ipxe/arm_Image initrd http://192.168.203.113/fog/service/ipxe/arm_init.cpio.gz append ...
            

            Step 😧 Testing Automatic Boot on the Raspberry Pi

             Used the U-Boot sequence with wget and the pxe boot command to attempt dynamic file retrieval at startup.
            
             Technical observation noted: The wget tool embedded in the BusyBox/FOS environment on ARM encounters parsing issues with URLs containing GET parameters (?mac=...), throwing a wget: bad port error.
            
          3. Attention Point

            The script generated by boot.php returns a Syslinux/PXE-style configuration format (label fog …), which is natively supported by U-Boot’s pxe boot command. However, full automation via a clean U-Boot command line (bootcmd) requires ensuring that the embedded wget on ARM environments can handle URLs with dynamic parameters without syntax errors.

          1 Reply Last reply Reply Quote 0
          • J
            Jeremy @Tom Elliott
            last edited by

            @Tom-Elliott

            Thanks for your clarification, and apologies for the confusion. I think there was a misunderstanding about what I am trying to achieve.

            I am not trying to bypass FOG or do something outside of FOG — quite the contrary. I am actively using FOG to deploy images to a fleet of Raspberry Pi devices.

            My previous confusion came from the fact that my test units kept landing in the interactive FOG debug mode (requiring pressing Enter to get a prompt), which led me down the wrong path thinking I needed a manual workaround. My ultimate goal is simply to have a standard, fully automated deployment task (non-debug) run seamlessly on the Raspberry Pis without requiring manual intervention at each boot.

            I will look into standard deployment tasks and post-init scripts as you suggested. Thanks for pointing me in the right direction!

            Tom ElliottT 1 Reply Last reply Reply Quote 0
            • Tom ElliottT
              Tom Elliott @Jeremy
              last edited by

              @Jeremy Two separate things here, and I think they have different answers.

              On landing in debug mode: worth being precise about what isdebug actually does, because it’s not what it might look like. The code that runs your postinit script (bin/fog) doesn’t check isdebug at all — it runs whenever fog executes, whether that’s automatic or you typed fog yourself at a shell. What isdebug=yes actually does is two things: it stops FOS from launching fog automatically at boot (you land at the debug info screen and then a bare shell instead), and it turns on every debugPause() — “Press [Enter] to continue” — checkpoint throughout the entire imaging engine, not just one place. There are dozens of them. So even if you manually ran your postinit script from the debug shell and it worked cleanly, anything FOG itself does afterward (a real deploy task) is going to stop and wait for Enter repeatedly, because isdebug=yes turns all of that on at once.

              That’s still coming from wherever you’ve got isdebug=yes set — either baked into your boot.cmd/append line, or as a Host Kernel Argument in the FOG web UI for this host, from when you added it to work around the multi-partition issue a few days ago. Clear it in both places. With it gone: fog runs automatically, your postinit script runs (same as before, that part was never conditional), the actual deploy runs with zero pauses, and the box reboots itself when the task completes — no interaction anywhere in the chain. That’s the automation you’re after.

              On the wget “bad port” error: worth being precise about where this is happening too — it’s U-Boot itself, at the wget … | pxe boot step, before FOS is even loaded. It isn’t BusyBox and it isn’t anything in FOS or the FOG server; the config I quoted came back correctly from curl, so the server side is fine.

              My best read — I don’t have your board to test this, so treat it as a hypothesis to check, not a diagnosis: U-Boot’s wget has two different implementations depending on how it was built. The newer one parses a real URL and splits host from path at the first /, so a query string with colons in it (like your MAC) shouldn’t confuse it. Older builds only understand wget <loadaddr> [<host-ip>:]<path> — no http://, no query-string awareness — and that parser looks for the first colon anywhere in the string, which lands inside mac=dc:a6:… and gets misread as a bogus port.

              Can you run wget with no arguments (or help wget) at the U-Boot prompt and paste the usage line it prints? That tells us which of the two you’ve got, and whether this is fixable on your end (newer U-Boot build) or needs the TFTP-staged fallback the docs mention.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              J 1 Reply Last reply Reply Quote 0
              • J
                Jeremy @Tom Elliott
                last edited by

                @Tom-Elliott

                I checked both database entries for hostKernelArgs, and isdebug=yes was indeed not present there.

                Regarding the U-Boot wget error, I followed your suggestion and went to the U-Boot prompt (=>) on the Raspberry Pi. When I type wget, it responds with:
                Unknown command wget try help

                So it seems the U-Boot build on these Raspberry Pis does not have the wget command compiled into it at all. What would be the recommended fallback or alternative method to fetch the script from FOG in this case (e.g., using TFTP or an alternative command if available)?"

                Tom ElliottT 1 Reply Last reply Reply Quote 0
                • Tom ElliottT
                  Tom Elliott @Jeremy
                  last edited by

                  @Jeremy Good, that narrows it down — no wget at all in this build, not just the older syntax. Two ways to close that:

                  Rebuild U-Boot with wget compiled in (keeps you on the dynamic per-host endpoint). CONFIG_CMD_WGET is a plain menuconfig option — you don’t need the HTTPS variant, which pulls in the whole lwIP/mbedTLS stack, just plain HTTP:

                  make rpi_4_defconfig
                  make menuconfig # Command line interface -> Network commands -> wget
                  make

                  Reflash the resulting u-boot.bin. Since dhcp already works for you, the network stack is there — this just turns on the one command. Once it’s in, the sequence from before should work as documented.

                  Or skip the dynamic endpoint and hand-roll a static boot.cmd — which is exactly what already worked for your capture: plain tftp for kernel/initrd/dtb, fixed bootargs with the task’s mac=, osid=, type=, img=, etc. baked in. The real tradeoff: tftp doesn’t ask FOG anything, so that config is identical for every board that boots it. Fine if every Pi runs the same task; not fine if you need FOG to decide per-host. boot.php exists specifically to answer “what does this MAC need right now” dynamically, and there’s currently no TFTP equivalent of it for boards that can’t do HTTP — that’s a real gap on FOG’s side, not something you’re doing wrong.

                  For 100+ boards I’d lean toward rebuilding U-Boot rather than going static, since you keep per-host tasking either way — but that’s your call based on how uniform the fleet’s deployment actually needs to be.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  J 1 Reply Last reply Reply Quote 0
                  • J
                    Jeremy @Tom Elliott
                    last edited by

                    @Tom-Elliott "Hi again,

                    Following your instructions, I checked the U-Boot prompt. When typing wget, it responds with:
                    Unknown command wget try help
                    So the U-Boot build on these boards does not include the wget command.

                    Also, for context on how the board behaves at boot, here is what happens visually during the startup sequence (attached photo): it tries to boot locally, times out on storage (mmc / usb), and then falls back to a standard network BOOTP broadcast loop (Retry time exceeded).

                    Given that U-Boot lacks wget and relies on standard BOOTP/PXE broadcast rather than direct HTTP fetching, what is the recommended way or fallback mechanism to have these boards pull their configuration from FOG?"

                    b1a36046-9e12-4cca-bb7a-8eeda752b04a-image.png

                    Tom ElliottT 1 Reply Last reply Reply Quote 0
                    • Tom ElliottT
                      Tom Elliott @Jeremy
                      last edited by

                      @Jeremy Good news on the wget front — I closed that gap I mentioned. FOG can
                      now serve boards with no wget at all, over plain TFTP instead of HTTP.

                      U-Boot’s pxe get command doesn’t need wget — every U-Boot build has it,
                      and unlike wget it isn’t configurable: it always fetches
                      pxelinux.cfg/01-<mac, dash-separated, lowercase> by TFTP from whatever
                      server your board’s serverip points at. FOG now keeps a real file at that
                      path for every host, kept in sync automatically whenever a task is queued,
                      completed, or canceled — so your boot sequence becomes just:

                      dhcp
                      pxe get
                      pxe boot
                      

                      No URL, no query string, nothing for the “bad port”/no-wget class of problem
                      to trip over — the filename is fixed by the MAC, so there’s nothing to
                      substitute.

                      This just merged into working-1.6 — same branch you’re already tracking
                      (you quoted 1.6.0-beta.4707 a few hours ago; this landed after that), so
                      running your update again should pick it up. Two things to check once you
                      have:

                      • Settings → FTP/TFTP needs a TFTP host/username/password configured — same
                        connection FOG already uses to upload your ARM kernel, so if that’s
                        already working for you, you’re set.
                      • I still don’t have a Pi to test this against, same as everything else in
                        this thread — if pxe get doesn’t find the file, or finds one FOG didn’t
                        mean to serve, that’s exactly the kind of thing I need you to report back.

                      One thing from your last post I want to flag separately, because it looks
                      like a different problem from the wget one: the boot sequence you
                      described — times out on local storage, then falls into a BOOTP broadcast
                      loop that gives up with “Retry time exceeded” — reads like U-Boot’s own
                      default bootcmd running, not a custom one. That’s the standard
                      Raspberry Pi fallback behavior when nothing else answers, and it wouldn’t
                      run dhcp / pxe get / pxe boot at all, custom or not.

                      Everything you’ve described testing so far — the wget/pxe boot sequence,
                      the curl check, now pxe get — sounds like it’s been typed by hand at the
                      => prompt each time. That proves the commands work, but it can’t be “fully
                      automated,” which was the goal from your very first post: nothing runs
                      automatically unless it’s persisted as bootcmd, either with
                      setenv bootcmd '...'; saveenv on the board itself, or as a boot.scr /
                      extlinux.conf your SD card or TFTP server hands U-Boot on its own at power-on.

                      Can you confirm which of those two you actually have in place right now? If
                      the answer is “neither yet — I’ve only been testing at the prompt,” that’s
                      probably the real blocker to automation, independent of wget vs. TFTP, and
                      worth sorting out before we look any further at the fetch mechanism itself.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      J 2 Replies Last reply Reply Quote 0
                      • J
                        Jeremy @Tom Elliott
                        last edited by

                        @Tom-Elliott

                        "To answer your question directly: yes, so far I have only been testing manually at the U-Boot prompt (=>).

                        My main concern right now is scalability. Since I need to deploy this across a fleet of 100 to 200 Raspberry Pi units, configuring each SD card manually or typing commands at the prompt for every single machine is not an option.

                        What is the recommended best practice in FOG working-1.6 to handle automation for a large fleet? Can the bootcmd be permanently stored in the Pi’s onboard EEPROM once, or can U-Boot fetch a boot script via TFTP automatically without manual card preparation?"

                        1 Reply Last reply Reply Quote 0
                        • J
                          Jeremy @Tom Elliott
                          last edited by

                          @Tom-Elliott

                          "Hi again! Great news regarding the new TFTP fallback mechanism.

                          I tested it manually by setting the IP addresses:
                          Plaintext

                          setenv ipaddr 192.168.203.50
                          setenv serverip 192.168.203.113
                          pxe get

                          What happened:

                          U-Boot successfully found the correct path generated by FOG based on the MAC address (Retrieving file: pxelinux.cfg/01-88-a2-9e-53-34-c0).
                          
                          However, the download times out (Loading: T T T T T T) and results in a Retry count exceeded.
                          

                          For context: a deployment task was explicitly active and queued in the FOG web interface before running the test, so the file should be there.

                          Is there a specific TFTP block size or setting required in working-1.6 for U-Boot’s pxe get to successfully pull the file without timing out, or could it be related to firewall/TFTP service configuration on the server side?"

                          Tom ElliottT 2 Replies Last reply Reply Quote 0
                          • Tom ElliottT
                            Tom Elliott @Jeremy
                            last edited by

                            @Jeremy That’s real progress — pxe get finding pxelinux.cfg/01-88-a2-9e-53-34-c0 by MAC means the naming convention and the file-generation side both work end to end. That’s the part I couldn’t test myself, so thank you for actually running it.

                            On the timeout itself: “found the file, then hung mid-transfer” (Retrieving file: ... succeeding, then Loading: T T T T T T / Retry count exceeded) points at TFTP’s own connection model, not at FOG. TFTP’s initial request goes to port 69, but the server then hands the rest of that transfer off to a different, randomly chosen UDP port for the actual data — so if there’s a firewall between the Pi and your FOG server that only allows port 69 through, the request succeeds (that’s why U-Boot found the file) and every packet after it gets silently dropped. That matches your symptom closely enough that I’d check it first: either open the OS’s ephemeral UDP range to your TFTP daemon, or (simpler if this is on the same box) confirm nothing’s filtering loopback/LAN traffic to it at all. If you’ve got shell on the FOG server, tcpdump -ni <iface> port 69 or portrange 32768-60999 while you retry pxe get should show the request going out and then nothing coming back, which would confirm it.

                            Separately — while looking into this I found a real gap on FOG’s side and fixed it regardless of whether it’s your actual cause: the file pxe get downloads wasn’t having its permissions set after upload, so if your TFTP daemon reads as a different user than the account FOG uploads over SFTP with, that alone could produce exactly this symptom (found the file, denied reading it). That fix is in the same working-1.6 branch now, so another update will pick it up. Worth ruling both things out — they’re not mutually exclusive.

                            On your scalability question: for 100–200 units, don’t script the SD card at all — the RPi 4’s own boot EEPROM supports network boot as the first boot mode, ahead of SD/USB, which is exactly what you want instead of the local-storage-timeout-then-BOOTP-fallback sequence you saw. rpi-eeprom-config (or the raspi-config boot order menu) lets you set that once, and once it’s set it’s a firmware-level property of the board, not something FOG or an SD card image controls — one flash per unit, and after that every power-on goes straight to network boot, no bootcmd persistence step needed on your end at all. I’d flash that as part of whatever imaging/provisioning process gets each Pi ready before it ever talks to FOG the first time, rather than trying to solve it per-deployment.

                            I don’t have a Pi 4 here to confirm the exact EEPROM boot-order syntax for your firmware revision, so double-check against Raspberry Pi’s own EEPROM config docs before you commit to a fleet-wide flash — but the “make network boot the default at the firmware level” shape of the answer is what actually gets you out of typing anything at the prompt.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                            1 Reply Last reply Reply Quote 0
                            • Tom ElliottT
                              Tom Elliott @Jeremy
                              last edited by

                              @Jeremy Correction on the firewall angle: if your server went through FOG’s own installer on ufw or firewalld, that’s actually less likely to be it than I made it sound — the installer already handles the exact “TFTP works for the request but the reply is on a random port” problem for both of those (a named tftp service on firewalld, and nf_conntrack_tftp loaded for ufw), specifically because it’s bitten people before.

                              So rather than “check your firewall” in general, four narrower things:

                              1. Which firewall are you actually running — ufw, firewalld, or plain iptables? The installer only auto-configures the first two; if it’s bare iptables, my original theory stands and you’d need to allow the ephemeral UDP range by hand.
                              2. If it’s ufw: lsmod | grep tftp — the installer tries to load nf_conntrack_tftp but silently continues if that fails, so it’s worth confirming it’s actually loaded rather than assumed.
                              3. Is BOOT_external_tftp_server set to “yes” in your FOG settings? If so and your TFTP is actually running on the FOG box itself, firewalld would have skipped opening it for you.
                              4. Are the Pi and the FOG server on the same subnet/VLAN, or is there a router, switch ACL, or (if this is a VM/cloud box) a security group between them? None of the above touches anything outside this box.

                              If you’ve got shell on the FOG server, tcpdump -ni <iface> port 69 or portrange 32768-60999 while you retry pxe get is still the fastest way to see whether the request goes out and nothing comes back (firewall/network) versus the request itself failing (something else entirely).

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              J 1 Reply Last reply Reply Quote 0
                              • J
                                Jeremy @Tom Elliott
                                last edited by

                                @Tom-Elliott

                                Here is an update following our latest troubleshooting sessions and tests:

                                The !V@JL artifact: That part was indeed a leftover artifact from editing the file manually with nano. Once cleaned up, we hit a regression: the Raspberry Pi no longer downloads the 563-byte config file at all anymore. It now loops directly on BOOTP broadcast and times out without hitting the TFTP stage (Retrieving file... doesn't trigger anymore).
                                
                                Goal & Architecture: The target deployment must remain 100% cardless (no local storage, no SD card, no USB drive used during boot), relying purely on the RPi 4 network boot EEPROM + OPNsense DHCP + FOG/U-Boot.
                                
                                Regarding your earlier notes on TFTP/Firewall:
                                
                                    We are running UFW on the FOG server, and we verified that nf_conntrack_tftp is active (lsmod | grep tftp confirmed).
                                
                                    BOOT_external_tftp_server is not enabled (TFTP runs locally on the FOG box).
                                
                                    The Pi and FOG server are on the same subnet, with no intermediate restrictive ACLs on that local segment.
                                
                                The permission fix: We noted your mention about the file permissions fix for downloaded files not being set properly after upload in the working-1.6 branch. Could this regression (dropping back to infinite BOOTP loops instead of attempting the TFTP fetch) be tied to how the file is currently served or indexed, or is there a specific way U-Boot expects the handoff after DHCP?
                                

                                4052d670-f8f4-4f76-ae50-03d630c809ec-image.png

                                Tom ElliottT 1 Reply Last reply Reply Quote 0
                                • Tom ElliottT
                                  Tom Elliott @Jeremy
                                  last edited by

                                  @Jeremy Good — that rules out the firewall/permissions angle cleanly, thanks for actually checking rather than taking my word for it.

                                  The new symptom is a different failure than before, and it’s outside anything FOG serves: looping on BOOTP with no attempt at Retrieving file... means it’s failing at DHCP/BOOTP negotiation, a step before pxe get would ever run. FOG can’t see that far back — the board never gets far enough to ask FOG anything.

                                  You’ve also moved to a materially different boot path than what we’d been testing: cardless, via the Pi’s own network-boot EEPROM, rather than typing commands by hand at the U-Boot prompt. That matters here, because the EEPROM’s automatic network-boot flow runs its own boot sequence, built into that U-Boot, not necessarily the dhcp / pxe get / pxe boot lines from earlier in this thread — those were for a manually-typed bootcmd. If the EEPROM path uses bootp instead of dhcp, or expects OPNsense to hand it specific DHCP options, that’s a different thing to get right than what we tested manually.

                                  Two things I’d need to actually say anything useful here, since I can’t see either from where I’m sitting:

                                  1. What’s the board actually running right now — did the nano edit touch a persisted bootcmd (printenv bootcmd), or is this the EEPROM’s own default network-boot sequence with nothing custom in the loop at all?
                                  2. In OPNsense’s DHCP config for that subnet, what are options 66 (next-server) and 67 (filename / bootfile-name) set to? The EEPROM’s PXE client needs those to know where to send its own request in the first place — if they’re pointing at the wrong place, or missing, U-Boot never gets the chance to see FOG at all, and that would produce exactly a BOOTP loop with nothing after it.

                                  That’ll tell us whether this is a boot-script problem (fixable on your end) or a DHCP-options problem (fixable in OPNsense) — right now I genuinely can’t tell which from here.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  J 1 Reply Last reply Reply Quote 0
                                  • J
                                    Jeremy @Tom Elliott
                                    last edited by

                                    @Tom-Elliott

                                    Hi, I’m sending you the two screenshots—one from OPNsense and the other from the Raspberry Pi.

                                    1f3ee7f3-69f9-473e-b150-4ff087aba3ba-image.png

                                    ec7c7fc7-dea7-4a5b-8ba0-6e22440ae87a-image.png

                                    Tom ElliottT 1 Reply Last reply Reply Quote 0
                                    • Tom ElliottT
                                      Tom Elliott @Jeremy
                                      last edited by

                                      @Jeremy Thanks for the screenshots — they changed my read of this quite a bit, and I owe you a correction first.

                                      Correction: earlier I said pxe get “finding” pxelinux.cfg/01-88-a2-9e-53-34-c0 proved the file-generation side worked. It didn’t. U-Boot prints Retrieving file: ... before it sends the request, and the Loading: T T T T that follows means the server never answered at all. A file that really is missing gives you TFTP error: 'File not found', not timeouts. So the file-side was never proven, and it turns out it was broken on my side of the fence.

                                      The FOG bug, now fixed: FOG was writing the 01-<mac> file under the directory it keeps the HTTP-served kernels in (/var/www/html/fog/service/ipxe/), not under /tftpboot. The TFTP daemon runs chrooted to /tftpboot, so the file existed but TFTP could never see it. That’s fixed in working-1.6 (PR #1664) with a new setting, FOG Settings → TFTP Server → FOG_TFTP_ROOT_DIR, which the installer sets to the real TFTP root. Run the updater once it’s merged and check that setting reads /tftpboot; after that, queue a task for the Pi and you should see /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 appear on the server. Please confirm that file is there before the next boot test — it’s the one thing I can check in code but not on your box.

                                      Your current boot loop is a different, earlier step. From the console shot: U-Boot itself loaded cardless, which means the Pi’s EEPROM network boot already pulled the firmware and u-boot.bin from /tftpboot over TFTP. That proves the OPNsense next-server, the FOG TFTP service and the firewall are all fine for this client — so ignore my earlier firewall angle. What’s looping is bootcmd=bootflow scan: that’s U-Boot’s standard-boot sequence, which tries mmc, usb, then ethernet, and the ethernet step starts with U-Boot’s own DHCP request (BOOTP broadcast 1, 2, 3...). That DHCP never succeeds, and it’s never been shown to work in this thread — your earlier manual test set ipaddr/serverip by hand and skipped DHCP entirely. The empty per-architecture filenames in OPNsense are fine, by the way: the pxe boot method doesn’t need a bootfile name at all. (Side note, unrelated to the Pi: the iPXE-class filename in OPNsense is what gets handed to a client that is already running iPXE, and FOG expects default.ipxe there, not ipxe.efi — with ipxe.efi your x86 UEFI clients will reload iPXE forever. Worth a look when you’re back on those.)

                                      Two things to try at the U-Boot prompt, and please paste the exact text rather than a screenshot if you can:

                                      dhcp
                                      

                                      on its own. If it also loops on BOOTP, then U-Boot’s network driver isn’t getting a lease and we look at OPNsense’s DHCP log for that MAC (does the request even arrive, does it offer). If it does get an address, then:

                                      pxe get
                                      pxe boot
                                      

                                      once the FOG update above is in and the 01-... file is confirmed on disk. If that works by hand, setenv bootcmd 'dhcp; pxe get; pxe boot' and saveenv gets you the automated path — bootflow scan should get there too, but the manual sequence tells us which step is at fault when it doesn’t.

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                      J 1 Reply Last reply Reply Quote 0
                                      • J
                                        Jeremy @Tom Elliott
                                        last edited by

                                        @Tom-Elliott

                                        I do have the filename you requested.

                                        c010fbb3-ed26-49e5-9a10-a3dc2a65bbd6-image.png

                                        When U-Boot starts, DHCP doesn’t work; it spends 8 seconds searching. Sometimes, when rebooting the Raspberry Pi, it manages to obtain an IP address, successfully detects the server’s IP, and downloads pxelinux, but then stops.

                                        b2f58c32-efc6-4017-9104-eafdf783d940-image.png

                                        What I don’t understand is why U-Boot sometimes manages to get an IP address and sometimes doesn’t.

                                        Tom ElliottT 2 Replies Last reply Reply Quote 0
                                        • Tom ElliottT
                                          Tom Elliott @Jeremy
                                          last edited by

                                          @Jeremy Do you happen to have STP on your network?

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                          1 Reply Last reply Reply Quote 0
                                          • Tom ElliottT
                                            Tom Elliott @Jeremy
                                            last edited by

                                            @Jeremy Good — the file being there confirms the /tftpboot fix landed. Two separate things in your report, and one of them was mine again.

                                            “Downloads pxelinux, but then stops” — partly expected, partly another FOG bug, now fixed.

                                            • If there was no task queued for the Pi at that moment, the file FOG writes says localboot 0, which tells U-Boot “nothing to do, carry on to local disk”. That’s by design: no task, no imaging. So “stops” after a successful pxe get is correct behavior unless you’d queued a capture first.
                                            • If there was a task queued, it would still have stopped, and that one is on me: the file named the kernel and init as http://… URLs, which is what boards with wget use. Your U-Boot’s pxe code can’t follow a URL — I checked U-Boot’s source (boot/pxe_utils.c😞 every kernel and initrd line is fetched over TFTP, relative to wherever the config came from, so kernel http://… became a TFTP request for a file literally called that. Fixed in working-1.6 (PR #1679): the file now says kernel arm_Image / initrd arm_init.cpio.gz, and FOG copies those two files into /tftpboot itself the first time a task is queued for an ARM host, and again whenever the kernel is updated. Run the updater once more, queue a capture for the Pi, then on the server check that /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 contains kernel arm_Image and that /tftpboot/arm_Image and /tftpboot/arm_init.cpio.gz exist.

                                            The intermittent DHCP is a network-timing problem, not a FOG one, and Tom’s STP question is the right first suspect. On a switch port running classic spanning tree, the port doesn’t forward traffic for roughly 30 seconds after link-up (listening, then learning). U-Boot brings the link up and sends its DHCP discover immediately, retries for a few seconds, and gives up — that’s the “8 seconds searching”. The times it works are the times the port happened to already be forwarding. The Pi’s EEPROM boot succeeds because it retries for much longer. Two fixes, and I’d do both:

                                            • On the switch: enable PortFast / edge-port (or RSTP) on the ports the Pis plug into. That’s the real fix and it helps every PXE client, not just the Pis.
                                            • In U-Boot, so a slow port doesn’t kill the boot anyway:
                                            setenv autoload no
                                            setenv netretry yes
                                            setenv bootcmd 'dhcp; pxe get; pxe boot'
                                            saveenv
                                            

                                            autoload no stops dhcp from also trying to TFTP a bootfile it was never given (that’s an extra timeout and a spurious failure in your sequence). netretry yes makes U-Boot keep retrying DHCP instead of giving up after a few seconds — the right call for a headless board that has nothing else to do, but be aware it means a Pi with no DHCP server on the wire will sit there retrying rather than dropping to the prompt.

                                            If DHCP still fails after that, the next read is OPNsense’s DHCP log for that MAC: whether the discover even arrives, and what it offers. And as before, pasted text beats a screenshot — the exact lines around pxe boot are the ones I need next.

                                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                            J 2 Replies Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 2 / 3
                                            • First post
                                              Last post

                                            46

                                            Online

                                            12.8k

                                            Users

                                            17.6k

                                            Topics

                                            157.0k

                                            Posts
                                            Copyright © 2012-2026 FOG Project