I know I’ve been out of this for a little bit. I check in here or there, but just been extremely busy.

I don’t want to stop contributing, I just am taking time for myself after my workly duties.

I have to give a big gratitude and thanks for everyone here trying to help out whether by code, by helping the rest of the community, or documentation.

@Sebastian-Roth I know you’re busy but you’ve kept the project rolling even with the minimal availability you have. Thank you.
@george1421 I’m sure you’re busy, but I still see you posting and helping where possible and amenible. Thank you.
@Wayne-Workman I know you’re helping where you can as well. (Of course I can’t exactly post everybody because I’ve been busy and honestly not keeping up with the forums as much as I probably should.)

@everyone Thank you. Thank you for still believing in this project. We’re doing the best with what we have. Please understand in we’re lacking, it’s most likely unintentional. I know I’m just busy.

https://news.fogproject.org/fog-1-3-5-and-client-0-11-11-officially-released/

https://news.fogproject.org/fog-1-5-0-rc-11/

TLDR; Rerun the fog installer if you have lost “Database Connectivity” to your fog server, or run the ALTER USER syntax shown below.

So Ubuntu 16, among others I suppose, enable a “security updates” to be applied automatically as a “default” to things. Why, well it makes it simpler to ensure your Ubuntu systems are in compliance and patched for any potential exploits. This causes unknown and unexpected issues.

I figured it’d be a safe thing to express that there could be problems (as many of you have already experienced) that when these updates go up (with or without your knowledge) it can break functionality in unexpected and inopportune ways.

The quickest fix is to simply rerun the fog installer which should correct the problem.

As a note, it seems this problem is specific only when the mysql account is the 'root' user AND the password is blank.

The “fix” if you must do it manually is to open a terminal and obtain root:
Super (Windows Key) + T then sudo -i (in most cases).

From there, open mysql with mysql -u root

NOTE: MySQL MUST be run with ROOT.

Run:

ALTER USER 'root'@'127.0.0.1' IDENTIFIED WITH mysql_native_password BY ''; AND
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '';

It’s okay if one of them fails. This is going to fix Most people’s issues.

I would highly recommend removing the unattended-upgrades as many of these “sudden” issues came as a security patch ubuntu pushed out. By default Ubuntu typically set’s this for you as enabled and it can cause havoc on you as you (the admin) may not have “done” anything.

To prevent this problem from happening in the future you could run:

apt-get -y remove unattended-upgrades (AS Root again).

FOG is still actively being developed. It’s not necessarily readily apparent, but we can assure you things are still being worked on. These updates may not be communicated in a way that everybody just knows, but can easily be seen if one were to look at our repository site.

Between our own schedules and lives, we can get very busy. We try to keep things updated and help out on the forums even during lull periods. This might mean we aren’t pushing an RC or release as frequently. It may mean we’re working on other things for the project, such as can be seen if looking at our github site.

Our forums are heavily active, and this should point as an indicator to our “status” as well.

If anybody would like to see an increase in developers donating their time to making this free software, consider donating either with monetary support or by spending personal time to help with development.

FOG is an open source project - it’s even in the name. It is driven by people donating their time and resources. The releases of FOG revolve around when developers can spare a few hours throughout the week. Sometimes that will mean releases will be further, sometimes that will mean releases will be faster. That’s just the nature of our project, and many other open source projects.

Hey everybody,

I know you see me here on occasion from time to time. Life decisions have made it more difficult for me to do things I would normally be doing. Rest assured, I am still around, and while I’m not quite as active as I was in the past, it’s not because I don’t want to be.

I had to move, and as part of that I have none of my normal development stuff readily available. Part of the move made me not have a laptop, until today.

I need to setup my dev environment again, so it may take a little bit, but I will be back up.

https://news.fogproject.org/fog-1-5-10-41-officially-released/

While this maybe spur of the moment, it should officially release 1.5.10 with all relevant bug/security fixes encapsulated (among a few other features.)

This has brought a new methodology of releases in that bug/security releases should be done much more regularly.

dev-branch, historically, was a place where new development occurred (hence its name) but over the last couple of years or so it’s mainly been a bug/security thing, not really a true development approach.

This is OKAY, in my head. Why:

Well we have the master branch which is the ‘baseline’ of a verions. We have the dev-branch, which allows us to work on bugs/security issues. We have the working branches for what will eventually become master. Working is our “dev branch” but keeps proper seperation of things in my opinion.

Basically:
master -> basis for dev-branch (which merges into stable on a regular cadence - still being worked out)
working -> basis of forward development

So we effectively have:
production
staging
development well seperated.

THis may mean on the regular automated releases, no announcement or news article will be created, and I think that’s okay.

Hopefully exciting we have a release after 1.25 years

Thank you!

That’s correct. The main reason fog is constantly moving forward is because the codebase is improved upon. Major bugs tend to be addressed for the next release. We don’t do an LTS because there’s really two main people working on fog in a consistent manor. Those two are @Joe-Schmitt and myself. Debian and Libreoffice have the team too be able to perform such a feat. Their product is Opensource but they have an employment team which can afford them that luxury. FOG has a team but we make no money and as such are required to work full time jobs. We work on FOG in our free time. I’ve had the ability to even work on it from work because we used the software.

Maintaining many different versions is difficult. And we don’t have a support team. WYSIWYG and I think we’ve done pretty well on support, even if we don’t have the ability to do dedicated support for our product. 1.5 was a major step toward modernizing the GUI. 1.6 will vastly improve on this. It was only recently we kind of came up with a road map on how best to proceed. Of note, 1.5 will be maintained until 1.6 is released. 1.6 is focused on making he GUI much more modern. 1.7 will be focused mostly toward fixing and refactoring the FOG client. 1.8 will focus on making the FOS system more modular and usable. I don’t know yet for 1.9. 2.0 will bridge the gap for our rewrite based on the work from 1.5 and up. While we do plan to try to do backports where possible, it’s much easier to ask people to update to the latest version than it is to try to maintain many different versions with backports in mind. At least for what FOG does.

I doubt this will appease anybody, but it’s what I think needs to be said. We are working hard and provide support for our product as best we can. The community makes fogs support system, I think, one of the best around. Add to that and you can almost always have a developer working side by side to help and fix issues as they come up, I don’t think it’s unfair to ask users to update to a specific version. Even if there are bugs, we will always try to correct what we can, when we can. (And normally it’s a pretty quick turn around).

I’m not perfect and I’ll give you that. We don’t even have a test suite to know if things are working as intended. We have to rely on the community and suggestions are great, just understand our answers won’t always be what people want to hear.

https://news.fogproject.org/fog-1-4-0-officially-released/

https://news.fogproject.org/fog-1-4-4-officially-released/

@ahaeder No:

The idea of fog client is to rename the host based on what you name the computer within FOG, not the other way around.

If you want it to be a specific name, and you have the FOG Client installed, set the wanted name in the FOG UI, and the host will change automatically.

@Jeremy Thanks, the file contents are the most useful thing you’ve posted, because they show the 01-… file on your server is not FOG’s. I owe you a correction on that too: yesterday I took “the file is there” as proof the fix landed. It wasn’t — that file is yours.

1. Everything in /tftpboot/pxelinux.cfg/ is hand-written, and it’s the wrong format for pxe boot. 01-88-a2-9e-53-34-c0, boot.scr, default and default-arm all contain a U-Boot script (setenv …, tftp …, booti …). pxe boot doesn’t run scripts; it parses a PXELINUX-style config (label, kernel, initrd, append), finds no labels in yours, and does nothing. That’s your “downloads pxelinux, but then stops”, and it was never going to work regardless of the FOG bugs I fixed. Please don’t hand-write anything under pxelinux.cfg/: FOG owns the 01-<mac> names there, and its periodic reconcile deletes any 01- file whose host has no active task, so your file would vanish anyway. (boot.scr, if you ever wanted one, has to be a mkimage-wrapped binary in the TFTP root, not text under pxelinux.cfg/. You don’t need it; the pxe path is the whole thing.)

To get FOG’s file: run the updater once more (PR #1679 is merged), then in the web UI queue a Capture task for Raspberry-test. Within a moment /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 will be replaced by a file starting # Generated by FOG Project with kernel arm_Image and initrd arm_init.cpio.gz, and /tftpboot/arm_Image and /tftpboot/arm_init.cpio.gz will appear beside it. Paste that file back here. Your host settings are fine: arm64, image set, primary disk /dev/sda. (acpi=off in the kernel arguments does nothing on a Pi; harmless.)

2. saveenv failed because there is no SD card, so nothing you setenv survives a reboot. That’s actually fine for the end goal: the default bootcmd=bootflow scan already does DHCP and then pxe get / pxe boot on its own, no environment needed. So for a cardless fleet you don’t need bootcmd at all. What you do need is DHCP to work inside U-Boot, which brings us to:

3. DHCP. netretry yes was doing its job (“Retry time exceeded; starting again” is it retrying, not failing). With STP confirmed on the switch, the port is most likely blocking for the first ~30 seconds after U-Boot resets the NIC. One test settles it: run setenv netretry yes then dhcp, and wait a full 60 seconds before judging. If it gets a lease on the second or third cycle, that’s STP, and the fix is PortFast / edge-port on the Pi ports (the default bootflow scan won’t retry long enough on its own, and without an SD card you can’t persist netretry). If it still has nothing after 60 seconds, then it isn’t timing: check OPNsense’s DHCP log (Services → DHCPv4 → Log) for 88:a2:9e:53:34:c0 during the attempt and tell me whether a DISCOVER arrives and whether an OFFER goes out.

Once DHCP works and FOG’s file is in place, bootflow scan (or dhcp; pxe get; pxe boot by hand) should pull arm_Image and boot FOS. If it stops anywhere after that, the last lines on screen are what I need — pasted text, not a photo, if there’s any way to capture it.

@Jeremy Good — the file being there confirms the /tftpboot fix landed. Two separate things in your report, and one of them was mine again.

“Downloads pxelinux, but then stops” — partly expected, partly another FOG bug, now fixed.

• If there was no task queued for the Pi at that moment, the file FOG writes says localboot 0, which tells U-Boot “nothing to do, carry on to local disk”. That’s by design: no task, no imaging. So “stops” after a successful pxe get is correct behavior unless you’d queued a capture first.
• If there was a task queued, it would still have stopped, and that one is on me: the file named the kernel and init as http://… URLs, which is what boards with wget use. Your U-Boot’s pxe code can’t follow a URL — I checked U-Boot’s source (boot/pxe_utils.c every kernel and initrd line is fetched over TFTP, relative to wherever the config came from, so kernel http://… became a TFTP request for a file literally called that. Fixed in working-1.6 (PR #1679): the file now says kernel arm_Image / initrd arm_init.cpio.gz, and FOG copies those two files into /tftpboot itself the first time a task is queued for an ARM host, and again whenever the kernel is updated. Run the updater once more, queue a capture for the Pi, then on the server check that /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 contains kernel arm_Image and that /tftpboot/arm_Image and /tftpboot/arm_init.cpio.gz exist.

The intermittent DHCP is a network-timing problem, not a FOG one, and Tom’s STP question is the right first suspect. On a switch port running classic spanning tree, the port doesn’t forward traffic for roughly 30 seconds after link-up (listening, then learning). U-Boot brings the link up and sends its DHCP discover immediately, retries for a few seconds, and gives up — that’s the “8 seconds searching”. The times it works are the times the port happened to already be forwarding. The Pi’s EEPROM boot succeeds because it retries for much longer. Two fixes, and I’d do both:

• On the switch: enable PortFast / edge-port (or RSTP) on the ports the Pis plug into. That’s the real fix and it helps every PXE client, not just the Pis.
• In U-Boot, so a slow port doesn’t kill the boot anyway:

setenv autoload no
setenv netretry yes
setenv bootcmd 'dhcp; pxe get; pxe boot'
saveenv

autoload no stops dhcp from also trying to TFTP a bootfile it was never given (that’s an extra timeout and a spurious failure in your sequence). netretry yes makes U-Boot keep retrying DHCP instead of giving up after a few seconds — the right call for a headless board that has nothing else to do, but be aware it means a Pi with no DHCP server on the wire will sit there retrying rather than dropping to the prompt.

If DHCP still fails after that, the next read is OPNsense’s DHCP log for that MAC: whether the discover even arrives, and what it offers. And as before, pasted text beats a screenshot — the exact lines around pxe boot are the ones I need next.

@Jeremy Do you happen to have STP on your network?

@Jeremy Thanks for the screenshots — they changed my read of this quite a bit, and I owe you a correction first.

Correction: earlier I said pxe get “finding” pxelinux.cfg/01-88-a2-9e-53-34-c0 proved the file-generation side worked. It didn’t. U-Boot prints Retrieving file: ... before it sends the request, and the Loading: T T T T that follows means the server never answered at all. A file that really is missing gives you TFTP error: 'File not found', not timeouts. So the file-side was never proven, and it turns out it was broken on my side of the fence.

The FOG bug, now fixed: FOG was writing the 01-<mac> file under the directory it keeps the HTTP-served kernels in (/var/www/html/fog/service/ipxe/), not under /tftpboot. The TFTP daemon runs chrooted to /tftpboot, so the file existed but TFTP could never see it. That’s fixed in working-1.6 (PR #1664) with a new setting, FOG Settings → TFTP Server → FOG_TFTP_ROOT_DIR, which the installer sets to the real TFTP root. Run the updater once it’s merged and check that setting reads /tftpboot; after that, queue a task for the Pi and you should see /tftpboot/pxelinux.cfg/01-88-a2-9e-53-34-c0 appear on the server. Please confirm that file is there before the next boot test — it’s the one thing I can check in code but not on your box.

Your current boot loop is a different, earlier step. From the console shot: U-Boot itself loaded cardless, which means the Pi’s EEPROM network boot already pulled the firmware and u-boot.bin from /tftpboot over TFTP. That proves the OPNsense next-server, the FOG TFTP service and the firewall are all fine for this client — so ignore my earlier firewall angle. What’s looping is bootcmd=bootflow scan: that’s U-Boot’s standard-boot sequence, which tries mmc, usb, then ethernet, and the ethernet step starts with U-Boot’s own DHCP request (BOOTP broadcast 1, 2, 3...). That DHCP never succeeds, and it’s never been shown to work in this thread — your earlier manual test set ipaddr/serverip by hand and skipped DHCP entirely. The empty per-architecture filenames in OPNsense are fine, by the way: the pxe boot method doesn’t need a bootfile name at all. (Side note, unrelated to the Pi: the iPXE-class filename in OPNsense is what gets handed to a client that is already running iPXE, and FOG expects default.ipxe there, not ipxe.efi — with ipxe.efi your x86 UEFI clients will reload iPXE forever. Worth a look when you’re back on those.)

Two things to try at the U-Boot prompt, and please paste the exact text rather than a screenshot if you can:

dhcp

on its own. If it also loops on BOOTP, then U-Boot’s network driver isn’t getting a lease and we look at OPNsense’s DHCP log for that MAC (does the request even arrive, does it offer). If it does get an address, then:

pxe get
pxe boot

once the FOG update above is in and the 01-... file is confirmed on disk. If that works by hand, setenv bootcmd 'dhcp; pxe get; pxe boot' and saveenv gets you the automated path — bootflow scan should get there too, but the manual sequence tells us which step is at fault when it doesn’t.

@azm9s UEFI cannot be disabled or do you mean “Secure Boot” cannot be disabled? UEFI disabled might make sense, Secure Boot not being able to be disabled, does not. Otherwise vendors would not be able to load custom Firmware or fix software.

Please review your MB for secure boot and disable it. I suspect that’s the issue you’re running into.

That said, what version of FOG are you running? You said a customized FOGProject, what exactly do you mean?

If you mean:

We used FOGProject as the baseline at some snapshot in time to do our own thing with it and we have it how we like it.

Then I don’t know what you want us to do. What you have is up to you to figure out how to get working because the developers would have no idea of what your layout looks like.

@Jeremy Good — that rules out the firewall/permissions angle cleanly, thanks for actually checking rather than taking my word for it.

The new symptom is a different failure than before, and it’s outside anything FOG serves: looping on BOOTP with no attempt at Retrieving file... means it’s failing at DHCP/BOOTP negotiation, a step before pxe get would ever run. FOG can’t see that far back — the board never gets far enough to ask FOG anything.

You’ve also moved to a materially different boot path than what we’d been testing: cardless, via the Pi’s own network-boot EEPROM, rather than typing commands by hand at the U-Boot prompt. That matters here, because the EEPROM’s automatic network-boot flow runs its own boot sequence, built into that U-Boot, not necessarily the dhcp / pxe get / pxe boot lines from earlier in this thread — those were for a manually-typed bootcmd. If the EEPROM path uses bootp instead of dhcp, or expects OPNsense to hand it specific DHCP options, that’s a different thing to get right than what we tested manually.

Two things I’d need to actually say anything useful here, since I can’t see either from where I’m sitting:

1. What’s the board actually running right now — did the nano edit touch a persisted bootcmd (printenv bootcmd), or is this the EEPROM’s own default network-boot sequence with nothing custom in the loop at all?
2. In OPNsense’s DHCP config for that subnet, what are options 66 (next-server) and 67 (filename / bootfile-name) set to? The EEPROM’s PXE client needs those to know where to send its own request in the first place — if they’re pointing at the wrong place, or missing, U-Boot never gets the chance to see FOG at all, and that would produce exactly a BOOTP loop with nothing after it.

That’ll tell us whether this is a boot-script problem (fixable on your end) or a DHCP-options problem (fixable in OPNsense) — right now I genuinely can’t tell which from here.

@Jeremy Correction on the firewall angle: if your server went through FOG’s own installer on ufw or firewalld, that’s actually less likely to be it than I made it sound — the installer already handles the exact “TFTP works for the request but the reply is on a random port” problem for both of those (a named tftp service on firewalld, and nf_conntrack_tftp loaded for ufw), specifically because it’s bitten people before.

So rather than “check your firewall” in general, four narrower things:

1. Which firewall are you actually running — ufw, firewalld, or plain iptables? The installer only auto-configures the first two; if it’s bare iptables, my original theory stands and you’d need to allow the ephemeral UDP range by hand.
2. If it’s ufw: lsmod | grep tftp — the installer tries to load nf_conntrack_tftp but silently continues if that fails, so it’s worth confirming it’s actually loaded rather than assumed.
3. Is BOOT_external_tftp_server set to “yes” in your FOG settings? If so and your TFTP is actually running on the FOG box itself, firewalld would have skipped opening it for you.
4. Are the Pi and the FOG server on the same subnet/VLAN, or is there a router, switch ACL, or (if this is a VM/cloud box) a security group between them? None of the above touches anything outside this box.

If you’ve got shell on the FOG server, tcpdump -ni <iface> port 69 or portrange 32768-60999 while you retry pxe get is still the fastest way to see whether the request goes out and nothing comes back (firewall/network) versus the request itself failing (something else entirely).

@Jeremy That’s real progress — pxe get finding pxelinux.cfg/01-88-a2-9e-53-34-c0 by MAC means the naming convention and the file-generation side both work end to end. That’s the part I couldn’t test myself, so thank you for actually running it.

On the timeout itself: “found the file, then hung mid-transfer” (Retrieving file: ... succeeding, then Loading: T T T T T T / Retry count exceeded) points at TFTP’s own connection model, not at FOG. TFTP’s initial request goes to port 69, but the server then hands the rest of that transfer off to a different, randomly chosen UDP port for the actual data — so if there’s a firewall between the Pi and your FOG server that only allows port 69 through, the request succeeds (that’s why U-Boot found the file) and every packet after it gets silently dropped. That matches your symptom closely enough that I’d check it first: either open the OS’s ephemeral UDP range to your TFTP daemon, or (simpler if this is on the same box) confirm nothing’s filtering loopback/LAN traffic to it at all. If you’ve got shell on the FOG server, tcpdump -ni <iface> port 69 or portrange 32768-60999 while you retry pxe get should show the request going out and then nothing coming back, which would confirm it.

Separately — while looking into this I found a real gap on FOG’s side and fixed it regardless of whether it’s your actual cause: the file pxe get downloads wasn’t having its permissions set after upload, so if your TFTP daemon reads as a different user than the account FOG uploads over SFTP with, that alone could produce exactly this symptom (found the file, denied reading it). That fix is in the same working-1.6 branch now, so another update will pick it up. Worth ruling both things out — they’re not mutually exclusive.

On your scalability question: for 100–200 units, don’t script the SD card at all — the RPi 4’s own boot EEPROM supports network boot as the first boot mode, ahead of SD/USB, which is exactly what you want instead of the local-storage-timeout-then-BOOTP-fallback sequence you saw. rpi-eeprom-config (or the raspi-config boot order menu) lets you set that once, and once it’s set it’s a firmware-level property of the board, not something FOG or an SD card image controls — one flash per unit, and after that every power-on goes straight to network boot, no bootcmd persistence step needed on your end at all. I’d flash that as part of whatever imaging/provisioning process gets each Pi ready before it ever talks to FOG the first time, rather than trying to solve it per-deployment.

I don’t have a Pi 4 here to confirm the exact EEPROM boot-order syntax for your firmware revision, so double-check against Raspberry Pi’s own EEPROM config docs before you commit to a fleet-wide flash — but the “make network boot the default at the firmware level” shape of the answer is what actually gets you out of typing anything at the prompt.

@Priyankha We need a lot more details.

FOG Version

What bootfile is your boot server trying to send?

Is you server sending the right FOG server for option 66/67?

What OS are you trying to capture?

What does the error show or maybe a screen shot? (UEFI Boot looks different from Legacy boot from Mac from arm, etc…)

What type of machine are you trying to image?

What have you tried?