@Tom-Elliott

I can’t modify the company’s switches or other hardware since the system is in production. I’m currently rebuilding the FOG server on a VM on my PC and doing everything locally; it’ll be easier to troubleshoot that way. Gemini has wiped out all the previous messages and is giving me nonsense—I can’t seem to recreate the environment up to the capture stage anymore. Could you give me a rundown of everything that needs to be done—downloads, decompressing specific files in binary mode, etc.?

Thanks.

@kratkale Two separate things. The first gets your PCs booting today.

PXE boot, now. The installer stopped before it rewrote the boot file, so the file still says https. Change it by hand:

sed -i 's#^chain https://#chain http://#' /tftpboot/default.ipxe grep chain /tftpboot/default.ipxe

The line must now start with chain http://192.168.0.196/. Boot one PC to test it. The next complete installer run writes this file again, with http.

The installer failure. Your certificates were not changed. The installer stopped before it created anything. The Secure Boot signing files that your settings name are not on disk, so it tried to create new ones. That needs the private key of your FOG root CA, and the key is not at /etc/fog/pki/root/ca/.fogCA.key. Your root certificate is still there, so your FOG clients are not affected.

Please do not delete anything, and do not run the installer with --recreate-CA. That replaces the CA that all your FOG clients trust.

Please post the output of these commands. They show only file names and paths, no key contents:

ls -la /etc/fog/pki /etc/fog/pki/root/ca /etc/fog/pki/secureboot /etc/fog/pki/secureboot/ca /etc/fog/pki/secureboot/leaf /opt/fog/snapins/ssl/CA ls -ld /opt/fog/pki grep -E '^PKI_(root|sb)_' /opt/fog/.fogsettings find / -xdev -name '.fogCA.key' 2>/dev/null

@Tom-Elliott Thank you, tested, confirmed it’s fixed.

@servicedesk-pianezza Yes, FOG writes that boot code, and I can show you where. Your finding holds up against our
source and against a reproduction here.

Where it comes from. On capture, saveGRUB() in funcs.sh copies the first 1 MiB of the
source disk into d1.mbr with dd — LBA0 included, boot code and all. On deploy,
clearPartitionTables() runs sgdisk -Z, which does clear the MBR, and then restoreGRUB()
writes d1.mbr straight back over it. The next two commands are sgdisk -z, which destroys
the GPT structures only and leaves the MBR bytes alone, and sgdisk -gl, which rewrites only
the protective partition entry. So the captured Windows bootstrap survives the whole
sequence, and gdisk supplies the entry in its own convention: EndCHS ff ff ff and the exact
sector count.

I replayed that sequence here on a loop-backed disk with one of our own Windows 11 resizable
images. The result matches what you found byte for byte in shape:

LBA0 : 33 c0 8e d0 bc 00 7c 8e ... (Windows MBR stub) 0x1BE : 00 00 02 00 ee ff ff ff 01 00 00 00 af 32 cf 1d

Why it is not simply a bug. On a BIOS-booted GPT Linux disk that same region is GRUB’s
boot.img, and d1.grub.mbr exists precisely so we keep it. We cannot blanket-zero LBA0 on
every GPT restore. On a Windows GPT image it is dead weight — Windows only boots GPT through
UEFI — so a targeted change is available to us. But we should change the right bytes.

Which byte is it? Your diskpart disk differs from ours in three places at once, so we do
not yet know which one the firmware chokes on. Three one-liners settle it. Start from a fresh
deploy that hangs, run one of them, power off, power on, press F2. Redeploy between tests so
each one is measured on its own:

# 1 - the bootstrap, nothing else dd if=/dev/zero of=/dev/nvme0n1 bs=1 count=446 conv=notrunc # 2 - the size field -> sentinel printf '\xff\xff\xff\xff' | dd of=/dev/nvme0n1 bs=1 seek=458 conv=notrunc # 3 - EndCHS -> the diskpart value printf '\xfe\xff\xff' | dd of=/dev/nvme0n1 bs=1 seek=451 conv=notrunc

Each writes inside LBA0 only and leaves the GPT untouched. Test 1 is the one I expect to
matter, on the theory you already stated — a CSM path in that firmware reading or validating
a bootstrap it should be ignoring.

Name the byte and the fix is small: for a GPT image whose OS is Windows, clear that field
during the restore and leave Linux images alone. That also gives the two 2020 reports on this
hardware an explanation, which is worth having on its own.

Now I read the AI summary and got what I need:

It stores one row for each address in the hostNetwork table. Limits of this method The sleeping host must run fog-agent. The server uses the sleeping host’s own last report to find its subnet. A host with the legacy FOG Client, or with no client, has no rows, so the relay cannot help it. The old path still runs for it.

So If I populate the hostNetwork table for the hosts that I need manually, It will work. Thank you for the details.

@ahaeder A final setting - you have to set Bios exit type = GRUB (not the default SANBOOT) or the boot hangs at ‘Booting from SAN device’.

@azm9s Good that legacy boot works now. For UEFI, your first tcpdump already shows one problem. It stops UEFI clients on every board, with Secure Boot on or off.

First: remove option 60 from your Windows DHCP server.
Your DHCP offer contains option 60 = “PXEClient”. UEFI firmware reads this as “this server is a PXE boot server”. It sends a second request to 192.168.65.35 on UDP port 4011 and waits. FOG does not listen on port 4011, so the firmware stops before it downloads any file. That is why no boot file worked. Legacy network boot does not do this, so legacy works. You only need option 60 if WDS runs on the same server as DHCP.

Second: set option 67 for UEFI to secureboot/snponly-shimx64.efi.
Your version (1.5.10.2253) installs this file. Check that it is there:

ls /tftpboot/secureboot/

It is signed by Microsoft, so it boots with Secure Boot on or off. You do not need to disable Secure Boot. Do not point option 67 at secureboot/ipxe.efi directly. With Secure Boot on, the firmware rejects it. If the menu loads but the network does not start, use secureboot/ipxe-shimx64.efi instead.

To keep legacy and UEFI working at the same time, use Windows DHCP policies. Vendor class PXEClient:Arch:00000 keeps your current legacy file, and PXEClient:Arch:00007 gets the file above:
https://docs.fogproject.org/en/latest/kb/how-tos/bios-and-uefi-co-existence

If UEFI still fails after these two changes, run this on the FOG server while the PC boots, and post the output:

tcpdump -i eth0 -n -vv ether host fc:9d:05:76:7c:00

@gmaurice resetting the host encryption in the gui and then restart the fog service and it should be back up and running. You can also use the api for this, the FogApi powerhsell module (links in my signature) I have this Reset-HostEncryption function https://fogapi.readthedocs.io/en/latest/commands/Reset-HostEncryption/?h=reset+host which will also handle this reset.

Your other other option is to look into post download scripts, there’s some examples in the forums and the docs. If you’re using sysprep and unattend.xml you can inject domain join information into the unattend.xml after imaging and before windows launches for the first time, so the computer is joined to the domain before the fog service or any ui is reachable.

@Balage80 Thanks for confirming the enrollment fix.

UEFI boot: I think the cause is two new lines in default.ipxe. They read Secure Boot state from the firmware. iPXE reads it by stepping through every firmware variable, and some firmware never ends that list, so iPXE hangs there.

Please test this: take the new 979-byte default.ipxe and delete only these two lines. Keep everything else.

param secureboot ${efi/SecureBoot} param setupmode ${efi/SetupMode}

Does UEFI boot with that file? Please also post the make, model, and BIOS version of the machine. Note that re-running installfog.sh writes a new default.ipxe, which replaces a manual edit.

Pending MACs: these are not related to default.ipxe. They come from the legacy FOG Client’s Host Registration module. That module reports every adapter Windows sees, including Wi-Fi, Bluetooth, and virtual Wi-Fi Direct adapters. FOG stores each unknown MAC as pending, up to FOG_QUICKREG_MAX_PENDING_MACS per host (default 4). iPXE cannot see those adapters. You can delete the pending MACs. To stop new ones, add MAC fragments to FOG_QUICKREG_PENDING_MAC_FILTER (comma separated), or turn off Host Registration.

@astrugatch I don’t think it matters really. The issue was more about the initial pinning of the certificate during the install process. After it is pinned I think this is the correct expectation (it should only communicate to the FOG server over HTTPS).

@Tom-Elliott I edited the sshd_config file and changed
/usr/lib/openssh/sftp-server
to
internal-sftp

I’m not sure how the installer failed to make this accomodation in the first place but I’m glad it appears to be working. I deployed the image to another laptop to make sure everything is working. It looks like the problem can be considered resolved now. Thanks for your help!

@jmeyer Would you be willing/able to try again without bypassing it to help confirm the fix?
@rogersk4132 thank you for testing and confirming!

@Tom-Elliott
Hi!
I have updated to the latest dev-branch version.
No more updating database … failed error.
The multicast session closing the right way.

Thank you!

Awesome and thank you. Confirmed working.

Hi Tom,

In my case the culprit was /images/postdownloadscripts/fog.postdownload.

echo "Activating boot partition..." sfdisk --activate /dev/sda 1 echo "Rebooting..." reboot -f

Fix was simply commenting out (or deleting) those lines. After cleanup the file looks like this:

cat /images/postdownloadscripts/fog.postdownload #!/bin/bash ## This file serves as a starting point to call your custom postimaging scripts. ## <SCRIPTNAME> should be changed to the script you're planning to use. ## Syntax of post download scripts are #. ${postdownpath}<SCRIPTNAME>

Thanks for the pointer, Tom — saved me from chasing GPT partition tables for a problem that had nothing to do with them.

Glad you have a workaround. I think the cause is the Windows boot manager certificate change, not the image.

Your golden Optiplex installed Windows with Secure Boot on. Windows servicing then added the “Windows UEFI CA 2023” certificate to that machine’s db, and switched the boot files to a boot manager signed with it. The other Optiplex 3000s only trust the 2011 Microsoft certificates, so they reject that boot manager. bcdboot works because it copies the older 2011-signed boot manager.

Can you confirm with two checks, in admin PowerShell, on the golden machine and on one target?

[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI db).bytes) -match 'Windows UEFI CA 2023' mountvol S: /s (Get-AuthenticodeSignature S:\EFI\Microsoft\Boot\bootmgfw.efi).SignerCertificate.Issuer

If the golden machine says True and the target says False, that is the cause. A newer Dell BIOS may include the 2023 certificate in its default keys. I am also looking at having FOS add it during the Secure Boot enrollment task.

@Tom-Elliott That was quick 😊 I just tested and it works as expected. Thank you.

Rahman

FYI for anyone else running into this issue - I believe the problem was that I didn’t erase the /tftpboot images when I switched from http to https. I’m not really sure why that would matter, but when I did that, then re-ran the server reinstall and selected https, the images got but back in /tftpboot and my hosts didn’t see the mac error anymore.

@jmeyer I have some e16 g1s and I am not having the same issue. What version of fog are you running? Are there any other secure boot settings you have configured or maybe not configured? Can you get to the mok enroll with secure boot off? And then turn it back on?

@JJ-Fullmer @edvandro This should be fixed in dev-branch as well, but I agree with JJ here, please upgrade to working-1.6.