I know I’ve been out of this for a little bit. I check in here or there, but just been extremely busy.

I don’t want to stop contributing, I just am taking time for myself after my workly duties.

I have to give a big gratitude and thanks for everyone here trying to help out whether by code, by helping the rest of the community, or documentation.

@Sebastian-Roth I know you’re busy but you’ve kept the project rolling even with the minimal availability you have. Thank you.
@george1421 I’m sure you’re busy, but I still see you posting and helping where possible and amenible. Thank you.
@Wayne-Workman I know you’re helping where you can as well. (Of course I can’t exactly post everybody because I’ve been busy and honestly not keeping up with the forums as much as I probably should.)

@everyone Thank you. Thank you for still believing in this project. We’re doing the best with what we have. Please understand in we’re lacking, it’s most likely unintentional. I know I’m just busy.

https://news.fogproject.org/fog-1-3-5-and-client-0-11-11-officially-released/

https://news.fogproject.org/fog-1-5-0-rc-11/

TLDR; Rerun the fog installer if you have lost “Database Connectivity” to your fog server, or run the ALTER USER syntax shown below.

So Ubuntu 16, among others I suppose, enable a “security updates” to be applied automatically as a “default” to things. Why, well it makes it simpler to ensure your Ubuntu systems are in compliance and patched for any potential exploits. This causes unknown and unexpected issues.

I figured it’d be a safe thing to express that there could be problems (as many of you have already experienced) that when these updates go up (with or without your knowledge) it can break functionality in unexpected and inopportune ways.

The quickest fix is to simply rerun the fog installer which should correct the problem.

As a note, it seems this problem is specific only when the mysql account is the 'root' user AND the password is blank.

The “fix” if you must do it manually is to open a terminal and obtain root:
Super (Windows Key) + T then sudo -i (in most cases).

From there, open mysql with mysql -u root

NOTE: MySQL MUST be run with ROOT.

Run:

ALTER USER 'root'@'127.0.0.1' IDENTIFIED WITH mysql_native_password BY ''; AND
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '';

It’s okay if one of them fails. This is going to fix Most people’s issues.

I would highly recommend removing the unattended-upgrades as many of these “sudden” issues came as a security patch ubuntu pushed out. By default Ubuntu typically set’s this for you as enabled and it can cause havoc on you as you (the admin) may not have “done” anything.

To prevent this problem from happening in the future you could run:

apt-get -y remove unattended-upgrades (AS Root again).

FOG is still actively being developed. It’s not necessarily readily apparent, but we can assure you things are still being worked on. These updates may not be communicated in a way that everybody just knows, but can easily be seen if one were to look at our repository site.

Between our own schedules and lives, we can get very busy. We try to keep things updated and help out on the forums even during lull periods. This might mean we aren’t pushing an RC or release as frequently. It may mean we’re working on other things for the project, such as can be seen if looking at our github site.

Our forums are heavily active, and this should point as an indicator to our “status” as well.

If anybody would like to see an increase in developers donating their time to making this free software, consider donating either with monetary support or by spending personal time to help with development.

FOG is an open source project - it’s even in the name. It is driven by people donating their time and resources. The releases of FOG revolve around when developers can spare a few hours throughout the week. Sometimes that will mean releases will be further, sometimes that will mean releases will be faster. That’s just the nature of our project, and many other open source projects.

Hey everybody,

I know you see me here on occasion from time to time. Life decisions have made it more difficult for me to do things I would normally be doing. Rest assured, I am still around, and while I’m not quite as active as I was in the past, it’s not because I don’t want to be.

I had to move, and as part of that I have none of my normal development stuff readily available. Part of the move made me not have a laptop, until today.

I need to setup my dev environment again, so it may take a little bit, but I will be back up.

https://news.fogproject.org/fog-1-5-10-41-officially-released/

While this maybe spur of the moment, it should officially release 1.5.10 with all relevant bug/security fixes encapsulated (among a few other features.)

This has brought a new methodology of releases in that bug/security releases should be done much more regularly.

dev-branch, historically, was a place where new development occurred (hence its name) but over the last couple of years or so it’s mainly been a bug/security thing, not really a true development approach.

This is OKAY, in my head. Why:

Well we have the master branch which is the ‘baseline’ of a verions. We have the dev-branch, which allows us to work on bugs/security issues. We have the working branches for what will eventually become master. Working is our “dev branch” but keeps proper seperation of things in my opinion.

Basically:
master -> basis for dev-branch (which merges into stable on a regular cadence - still being worked out)
working -> basis of forward development

So we effectively have:
production
staging
development well seperated.

THis may mean on the regular automated releases, no announcement or news article will be created, and I think that’s okay.

Hopefully exciting we have a release after 1.25 years

Thank you!

That’s correct. The main reason fog is constantly moving forward is because the codebase is improved upon. Major bugs tend to be addressed for the next release. We don’t do an LTS because there’s really two main people working on fog in a consistent manor. Those two are @Joe-Schmitt and myself. Debian and Libreoffice have the team too be able to perform such a feat. Their product is Opensource but they have an employment team which can afford them that luxury. FOG has a team but we make no money and as such are required to work full time jobs. We work on FOG in our free time. I’ve had the ability to even work on it from work because we used the software.

Maintaining many different versions is difficult. And we don’t have a support team. WYSIWYG and I think we’ve done pretty well on support, even if we don’t have the ability to do dedicated support for our product. 1.5 was a major step toward modernizing the GUI. 1.6 will vastly improve on this. It was only recently we kind of came up with a road map on how best to proceed. Of note, 1.5 will be maintained until 1.6 is released. 1.6 is focused on making he GUI much more modern. 1.7 will be focused mostly toward fixing and refactoring the FOG client. 1.8 will focus on making the FOS system more modular and usable. I don’t know yet for 1.9. 2.0 will bridge the gap for our rewrite based on the work from 1.5 and up. While we do plan to try to do backports where possible, it’s much easier to ask people to update to the latest version than it is to try to maintain many different versions with backports in mind. At least for what FOG does.

I doubt this will appease anybody, but it’s what I think needs to be said. We are working hard and provide support for our product as best we can. The community makes fogs support system, I think, one of the best around. Add to that and you can almost always have a developer working side by side to help and fix issues as they come up, I don’t think it’s unfair to ask users to update to a specific version. Even if there are bugs, we will always try to correct what we can, when we can. (And normally it’s a pretty quick turn around).

I’m not perfect and I’ll give you that. We don’t even have a test suite to know if things are working as intended. We have to rely on the community and suggestions are great, just understand our answers won’t always be what people want to hear.

https://news.fogproject.org/fog-1-4-0-officially-released/

https://news.fogproject.org/fog-1-4-4-officially-released/

@Jason89436 Thanks for the report. The agent had no snapin pack support, and on Windows it also dropped backslashes from snapin arguments. Both are fixed. Update the server to the latest working-1.6, then update the agent to 0.1.10. After that, [FOG_SNAPIN_PATH]\msoffice.ps1 resolves to the unzipped pack folder, as it did with the legacy client.

@mp12 Thanks for the logs. This is a bug in 1.5.10.2482, not your node passwords.

A security change in 2482 removes the storage node password from the node data that the API returns. The image and snapin replicators read their node list from that same data. So they now send an empty password, and every node rejects the login. The Undefined property: stdClass::$pass warning is that missing field.

The fix is merged to dev-branch: https://github.com/FOGProject/fogproject/pull/1770

To get it now, update from dev-branch:

cd /path/to/fogproject
git checkout dev-branch
git pull
cd bin
sudo ./installfog.sh -y

Or wait for the next stable release. Your stored passwords are correct, so you do not need to change anything on the nodes.

@rdr

How FOG wakes a host through the fog-agent wake relay

Summary

FOG wakes a host in two ways at the same time. First, the FOG server and the storage nodes send the magic packet on their own subnets. Second, if the relay is on, the server asks up to three awake agents on the host’s subnet to send the packet. The server always picks the target and the senders. An agent only sends.

The problem

A Wake-on-LAN (WoL) magic packet is a broadcast. A broadcast stays on one subnet. Before the relay, only the FOG server and the storage nodes sent the packet. So FOG could not wake a host on a subnet with no FOG server and no storage node.

To send a broadcast to a remote subnet (“directed broadcast”), the routers must forward it. Most networks disable that router feature, because attackers used it for amplification attacks. The wolbroadcast plugin depends on that feature.

The relay removes the gap. Every such subnet has FOG hosts on it. When one of them is awake, it can send the packet for its neighbor.

How FOG knows which hosts share a subnet

Each fog-agent reports its network interfaces to the server on every poll. For each interface, the report gives:

• the IPv4 address and the prefix length (for example, 10.1.5.23 and /24)
• whether the interface is up and has a link (a NIC with no cable is not up)
• whether the interface is wireless

The server does not trust a network address from the agent. It calculates the network address and the broadcast address itself, from the address and the prefix. It stores one row for each address in the hostNetwork table.

Example:

Host	Reported	Network the server calculates
Host 41 (asleep)	10.1.5.23/24	10.1.5.0/24
Host 77 (awake)	10.1.5.80/24	10.1.5.0/24
Host 90 (awake)	10.1.0.12/16	10.1.0.0/16

Hosts 41 and 77 share a subnet: the network address and the prefix are both equal. Host 90 does not share it. Its network address is different, and a /16 and a /24 are never one subnet.

To find senders, the server does one database lookup. It takes the rows of the sleeping host, and it finds other hosts with the same network address and the same prefix. It then keeps only the hosts that meet all of these conditions:

• The interface is up and has a link.
• The interface has a broadcast address. A /31 or /32 link has none.
• The interface is not wireless. An access point does not pass a broadcast to a machine that is asleep, because that machine is no longer connected to the access point.
• The host polled in the last 900 seconds, so it is probably awake.
• The host is not the sleeping host.

The server sorts the result by the most recent poll and keeps the first three.

Limits of this method

• The sleeping host must run fog-agent. The server uses the sleeping host’s own last report to find its subnet. A host with the legacy FOG Client, or with no client, has no rows, so the relay cannot help it. The old path still runs for it.
• The sleeping host’s subnet is its last report. A laptop that moved to a different subnet while off is looked for on the old subnet.
• Only IPv4. A magic packet uses an IPv4 broadcast.

The flow, step by step

Step: someone asks for a wake. The source is the Wake Up button, a group wake, or a scheduled wol task. Each of these calls Host::wakeOnLAN().

Step: the old path runs first, and it does not change. The server sends a request to every storage node and to itself. Each of them sends the magic packet on its own subnets. If a storage node shares the host’s subnet, this path is enough.

Step: the relay path runs as an addition. It does nothing unless the global setting FOG_AGENT_WAKE_RELAY_ENABLED is 1. The default is 0. The server finds up to three senders, as described above. It writes one agentWake row for each pair of sleeping host and sender. Each row expires after 600 seconds.

Step: the sender agent receives the request on its next poll. The agent does not listen on a network port. The request is part of the normal poll answer:

"wake": {"targets": [{"id": 41, "macs": ["00:11:22:33:44:55"]}]}

The block contains no destination address. It contains only the host id and the MACs of that host. The server leaves out pending MACs that nobody approved.

Step: the agent sends the packet.

• The agent parses each MAC and builds it again. It refuses a MAC that is not valid.
• It builds the 102-byte magic packet.
• It sends the packet to UDP port 9, at 255.255.255.255 and at the broadcast address of each of its own interfaces.
• It sends to at most 32 hosts per poll, and at most one packet for each MAC on each interface.

Step: the agent reports the result. The result is sent with a packet count, or failed with a reason. The server accepts the result only if a pending agentWake row names this sender and this sleeping host. Otherwise it returns 404. So an agent cannot report on a host that the server did not ask it to wake.

Why the design has this shape

Choice	Reason
The server picks the target and the senders	A magic packet has no authentication. The control must be on who can ask. Only the server knows which hosts are real FOG hosts.
The request is part of the poll, with no network port on the agent	An open port lets anyone who can reach it ask for a broadcast.
The request has no address field	An agent that accepts a destination can be used to send traffic at any address.
Three senders, not one	Extra packets cost almost nothing. With one sender, the wake fails silently if that sender goes to sleep.
Requests expire after 600 seconds	A laptop that comes back next week must not send an old wake.
Off by default	One customer machine sends traffic for another. The estate owner must choose that.
Wireless interfaces are never senders	The access point does not deliver the broadcast to a sleeping machine.

The cost: delay

A relayed wake goes out on the sender’s next poll. With the default interval, that is up to 5 minutes. Most WoL use in FOG is a scheduled overnight task, so the delay is acceptable. The old path still sends immediately.

What the relay is not

• It cannot wake a device that FOG does not manage.
• It does not replace the storage-node path or the wolbroadcast plugin.
• Agents do not talk to each other. Every decision is the server’s.
• It does not wake a host across the internet. A magic packet stays on one subnet.

Sources

• Design: fog-agent/docs/design/0011-wake-relay.md
• Wire format: fog-agent/docs/design/protocol-v1.md, section “Wake”
• Server: packages/web/src/Agent/WakeRelay.php (senders()), packages/web/src/Agent/NetworkFacts.php, packages/web/src/Items/Host.php (wakeOnLAN())
• Agent: internal/network/network.go, internal/provider/wake/

@rdr I think I don’t fully understand either.

Basically, behind the scenes as I understand the build:

FOG Server wants to wake a machine. FOG Server tries to send a Magic packet to the MAC address in question. It also checks the ARP table of the network to see what FOG agents might know about the MAC being requested to wake, and if an agent is alive on the same subnet that MAC lives on, it will ask the FOG Agent to also try to wake it up.

So I don’t think it knows it the way we are thinking about things. It’s a bit more nuanced than that, but that’s highly suspected from my understanding of things, not necessarily exactly how it knows.

I’ll ask claude code to see if it can give me a quick rundown of how it works from a design and flow perspective.

@GRISLET The task is only marked complete by one request: POST /fog/service/Post_Stage3.php, sent by fog.imgcomplete as the last step of the deploy. Your log shows it never went out, so the script exited before reaching it.

You see no error because S99fog prints * Rebooting system as task is complete and reboots whenever /bin/fog exits, for any reason. A silent early exit is indistinguishable on screen from a real completion.

The 48-second gap points at where. fog.statusreporter posts progress.php every 3 seconds for the whole task and stops only when killStatusReporter kills it — which is the first line of completeTasking. After that line, only three things run before the completion POST. One of them is /images/postdownloadscripts/fog.postdownload, which is sourced into the imaging shell. An exit or a reboot in that script, or in any script it calls with ., ends the task before FOG is told about it.

Two things would confirm it:

1. cat /images/postdownloadscripts/fog.postdownload, plus any script it sources. Look for exit or reboot.
2. The last ten lines on the client screen before the reboot. Do Stopping FOG Status Reporter, * Task Complete and Updating Database appear? If they do not, the run ended early and the image type is not involved.

The image type is probably a red herring. Nothing between the end of the restore and the completion POST depends on Single Disk - Resizable.

@AUTH-IT-Center I do believe snponly would be the recommended, rather than iPXE’s driver.

The developers at iPXE wrote the driver on their own (of course using documentation and stuff, but for all intents/purposes it is still a handrolled driver) so anything is possible.

We shipped the native iPXE 2.0.0 mainly because of the feature it allows with actual Secureboot capabilities and instead of embedding everyfile with a custom script, a more dynamic approach for when iPXE releases new version we can upgrade more easily.

For what it’s worth, I would almost want more people to default to snponly.efi (or secureboot/snponly-shimx64.efi if using/wanting secureboot after enrolling your machines of course) because this is supposed to be using the generic driver for EFI boot protocols on the NIC rather then attempting to discover the NIC using a driver loaded.

@rdr I don’t believe the FOG-Agent would be the right thing for this, and neither would the FOG Server.

In very old versions of FOG the IP address was a distinguishing factor of what a “host” was.

Relatively shortely after that became a norm, machines really started to come with multiple NICs, and/or with Wifi. That plus many systems prefering to move to DHCP for their networks (due to Wifi needs, and manual intervention to change an ethernet when a machine moved from one place to the other) FOG stopped trying to use the IP as a definition of what a host was.

WOL Works on macs and generally locally to the subnet the FOG server (or local machines) run on, but with some finagling you can send WOL packets on different network subnets.

Welcome the WOLBroadcast Plugin that already exists. I will admit I’ve not tested this plugin in quite some time now, so please see if that will be more what you’re looking for?

The FOG Agent shouldn’t be simply spamming your network with WOL packets just because it doesn’t know what is on the same subnet (to your initial point), but the FOG Server should be able to especially if you have your network already done for this.

I hope this helps with what you’re looking for. The FOG Agent, in my eyes, should only try to do things it knows about, instead of being a rogue “DDoS” vector.

@rdr there is a feature in the fog agent (although untested at this point) that can tell the fog-agent to try to send a WOL packets. If an agent machine is up, and you need to wol a machine that’s on the same subnet, it can happen. It is disabled by default.

I think that’s what you were attempting to ask? I hope I didn’t butcher it too badly.

https://docs.fogproject.org/en/latest/management/web/fog-agent#waking-a-host-on-a-subnet-with-no-fog-server

Glad you have a workaround. I think the cause is the Windows boot manager certificate change, not the image.

Your golden Optiplex installed Windows with Secure Boot on. Windows servicing then added the “Windows UEFI CA 2023” certificate to that machine’s db, and switched the boot files to a boot manager signed with it. The other Optiplex 3000s only trust the 2011 Microsoft certificates, so they reject that boot manager. bcdboot works because it copies the older 2011-signed boot manager.

Can you confirm with two checks, in admin PowerShell, on the golden machine and on one target?

[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI db).bytes) -match 'Windows UEFI CA 2023'

mountvol S: /s
(Get-AuthenticodeSignature S:\EFI\Microsoft\Boot\bootmgfw.efi).SignerCertificate.Issuer

If the golden machine says True and the target says False, that is the cause. A newer Dell BIOS may include the 2023 certificate in its default keys. I am also looking at having FOS add it during the Secure Boot enrollment task.

@Strahd Can you please get the output of the line:

grep -P 'Subsystem.*sftp' /etc/ssh/sshd_config

It should output something like:

Subsystem       sftp    internal-sftp

If it does not look like the above, edit the file (as root) look for the matched line and make it look like the above, then restart sshd service (systemctl restart sshd)

Thank you,