@JJ-Fullmer My version is 270. Do you still want me to try updating PHP to 8.3?

2b08c072-e93f-40b2-b2e2-43226ebf9d98-image.png

@Tom-Elliott said in Wiping Drive via PXE menu:

@vanfifty1 Sweet so it is functional and seems to be much more verbose about the actions it’s performing, thank you!

Thank you!

With the experimental init.xz when it said the following in my case: “Erasing /dev/nvme0n1 with an NVMe sanitize block erase (sanact 2)” does this mean the following command was run?: nvme sanitize /dev/nvme0n1 -a 2

Does this mean the bug is patched? Should I use the experimental init.xz file on the production FOG server? What ramifications will this have? Is it better to wait until this is released in an official version down the road?

@MatMurdock @Fog_Newb

Apologies, this one’s on me.

There’s a constant, FOG_BASE_DIR, that the Secure Boot signing code uses to find where FOG is installed. On 1.6 the installer writes that value out for PHP to read. When I ported the signing code back to 1.5 I brought the half that uses the constant and not the half that sets it. On PHP 7 that’s a warning nobody sees. On PHP 8 it’s a hard fatal, which is where the 500 comes from.

@MatMurdock thank you, you found exactly the right line and your patch is sound.

@Fog_Newb your bisect nailed it. 2144 good, 2191 bad, and the commit that broke it is 2179. That saved me a lot of digging.

The proper fix is merged and is in 1.5.10.2213 on dev-branch. Update and re-run the installer. I’ve confirmed the fatal is gone, but I haven’t been able to reproduce your setup end to end, so please let me know either way whether the kernel page behaves for you now. If it still 500s, grab the Apache error log or php-fpm log around the failure and post it, because then it’s something else and I want to see it.

Two things on the boot problem:

I still can’t reproduce the ram-0 error and I don’t want to guess at it. Can you get a photo of the screen when it fails? The exact wording matters. And let me know if that laptop is booting UEFI or legacy BIOS.

If it’s legacy BIOS, there’s a separate bug that was also live in your window. 2191 shipped an iPXE build with a batch of commands disabled that our boot menu needs. Symptom is params: command not found then Could not boot: Exec format error. That went in at 2208, but the server won’t heal itself, /tftpboot/undionly.kkpxe only gets replaced when the installer runs again.

Worth trying the ramdisk size too since it’s quick. FOG Configuration → FOG Settings → Kernel Ramdisk Size, bump it to 512000.

@JJ-Fullmer This was it, thank you!!

Everything I read kept telling me this machine was x86 based and there is no hint in the BIOS or the serial number lookup tool, of it running an ARM processor(other then “Elite”). We are going to return these machines to the vendor, so problem solved for us.

Would be interested in hearing about future support for ARM in FOG because I’m sure this isn’t the last time we will cross paths with it.

@Cpasjuste

Thank you, I appreciate that. And yes, I did spend a lot of time on it.

I agree that it bypasses the official installer in order to make use of Docker environment variables; however, at this stage, I don’t believe that FOG is going to change so dramatically that it’ll break it often.

I will add that if this could get adopted as an official docker image, we could work with the FOG team to make sure nothing gets to pushed to the FOG code that would break the container until we can adjust the container as well.

@Tom-Elliott no, i have bypass dhcp configuration during install.

To check the BitLocker status, open an elevated Command Prompt and run:

manage-bde -status

If the C drive shows as encrypted, disable BitLocker and allow the drive to fully decrypt before attempting another capture:

manage-bde -off C: manage-bde -off D:

Have you tried setting the image type to “Multiple Partition Image - All Disks (Not Resizable)” instead of “Single disk - resizable”?

In “Single disk - resizable” mode, you will save an image of only one of the two disks. In the host settings, you can set the primary disk.

Try disable Windows Fast Boot or restart computer from cmd

shutdown /r /t 0

@jmeyer Good work. So far looks very similar to what I was doing above but with grub instead. I’m guessing you’re aiming for something along the lines of the archived project you mentioned earlier that did this. Make sure you modify your ipxe scripts to load the shim with the shim command, and then sign whatever you’re booting from ipxe and you should be more or less there.

It’s worth noting if you want you should be able to skip the grub stage entirely, if you load the shim directly and name your ipxe binary what you’re grub binary currently is you should be able to net boot any shim pretty easily, from there the shim can automatically call mok manager as long as it’s in the same directory as your shim.

Sorry for responding so late. I’m out on training this week.

@toschulz So I suspect the problem is due to a divergence of db schema between dev-branch/stable + working-1.6

If you can update yoru Mysql directly:

sudo mariadb -u root fog (if no root password of course - if you have a root password add -p

UPDATE schemaVersion SET vVersion=0;

Then refresh your screen, click the install/update, and hopefully the error will go away.

The way the schema is configured is it will attempt (idempotently) to rerun every single SQL query in the schema chain.

This should fix the problem you’re seeing.

I believe there’s a column on the login history piece being requested that may just simply be missing which would cause the issue with datatables.

@DiegoP I believe mdadm will still be allowed to work, but I think you’re supossed to use “mdraid=true” for a kernel argument. If this is not set, FOG won’t know to build the array. This would help explain why the disk couldn’t find a partition layout as well since it’s expecting to be part of an array (which then looks/acts as the disk you’re truly working with.

As for “partclone to partimage” I’m not sure I understand.

Fog moved FROM partimage -> partclone way back in the 1.0.0 days as partimage has been obsolete for quite some time now.

@Tom-Elliott

Thanks for your quick response. We’ll be using Fog heavily over the next few days, so we won’t be able to update the version right away.
We’ve found a workaround while we wait for the update by setting the setgid bit on the directory:

chown fog:fog /images/dev
chmod 2775 /images/dev

Best.

Jérôme

@Tom-Elliott Hello Tom yes I have installed the compatible Fog client from the Fog server on the reference machine, sys prepped and captured the image and now I am deploying.

@Tom-Elliott
Updated to latest dev-branch and snapins are working now
Thansk for help.

Hi @Tom-Elliott – thank you for your help! The client is UEFI.

I probably should have mentioned that the main symptom I’m trying to figure out is the client giving up with PXE-E16: No valid offer received after a few minutes of trying PXE boot.

Interestingly, when I tried also listening to port 4011 with my initial config file, there was no additional traffic being picked up.

When I used the version you suggested (with the server’s IP switched in of course), the tcpdump actually no longer shows any responses to the client’s requests, and the PXE boot errors out faster with PXE-E18: Server response timeout.

tcpdump: listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 16:08:07.096920 IP (tos 0x0, ttl 64, id 29391, offset 0, flags [none], proto UDP (17), length 375) 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from <client’s mac address>(oui Unknown), length 347, xid 0x92c2c13c, Flags [Broadcast] Client-Ethernet-Address <client’s mac address>(oui Unknown) Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message (53), length 1: Discover MSZ (57), length 2: 1472 Parameter-Request (55), length 35: Subnet-Mask (1), Time-Zone (2), Default-Gateway (3), Time-Server (4) IEN-Name-Server (5), Domain-Name-Server (6), Hostname (12), BS (13) Domain-Name (15), RP (17), EP (18), RSZ (22) TTL (23), BR (28), YD (40), YS (41) NTP (42), Vendor-Option (43), Requested-IP (50), Lease-Time (51) Server-ID (54), RN (58), RB (59), Vendor-Class (60) TFTP (66), BF (67), GUID (97), Unknown (128) Unknown (129), Unknown (130), Unknown (131), Unknown (132) Unknown (133), Unknown (134), Unknown (135) GUID (97), length 17: 0.128.226.9.231.152.246.237.17.132.7.227.255.61.233.42.0 NDI (94), length 3: 1.3.16 ARCH (93), length 2: 7 Vendor-Class (60), length 32: "PXEClient:Arch:00007:UNDI:003016" 16:08:10.881261 IP (tos 0x0, ttl 64, id 29392, offset 0, flags [none], proto UDP (17), length 375) 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from <client’s mac address>(oui Unknown), length 347, xid 0x92c2c13c, secs 4, Flags [Broadcast] Client-Ethernet-Address <client’s mac address>(oui Unknown) Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message (53), length 1: Discover MSZ (57), length 2: 1472 Parameter-Request (55), length 35: Subnet-Mask (1), Time-Zone (2), Default-Gateway (3), Time-Server (4) IEN-Name-Server (5), Domain-Name-Server (6), Hostname (12), BS (13) Domain-Name (15), RP (17), EP (18), RSZ (22) TTL (23), BR (28), YD (40), YS (41) NTP (42), Vendor-Option (43), Requested-IP (50), Lease-Time (51) Server-ID (54), RN (58), RB (59), Vendor-Class (60) TFTP (66), BF (67), GUID (97), Unknown (128) Unknown (129), Unknown (130), Unknown (131), Unknown (132) Unknown (133), Unknown (134), Unknown (135) GUID (97), length 17: 0.128.226.9.231.152.246.237.17.132.7.227.255.61.233.42.0 NDI (94), length 3: 1.3.16 ARCH (93), length 2: 7 Vendor-Class (60), length 32: "PXEClient:Arch:00007:UNDI:003016" 16:08:18.968033 IP (tos 0x0, ttl 64, id 29393, offset 0, flags [none], proto UDP (17), length 375) 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from <client’s mac address>(oui Unknown), length 347, xid 0x92c2c13c, secs 12, Flags [Broadcast] Client-Ethernet-Address <client’s mac address>(oui Unknown) Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message (53), length 1: Discover MSZ (57), length 2: 1472 Parameter-Request (55), length 35: Subnet-Mask (1), Time-Zone (2), Default-Gateway (3), Time-Server (4) IEN-Name-Server (5), Domain-Name-Server (6), Hostname (12), BS (13) Domain-Name (15), RP (17), EP (18), RSZ (22) TTL (23), BR (28), YD (40), YS (41) NTP (42), Vendor-Option (43), Requested-IP (50), Lease-Time (51) Server-ID (54), RN (58), RB (59), Vendor-Class (60) TFTP (66), BF (67), GUID (97), Unknown (128) Unknown (129), Unknown (130), Unknown (131), Unknown (132) Unknown (133), Unknown (134), Unknown (135) GUID (97), length 17: 0.128.226.9.231.152.246.237.17.132.7.227.255.61.233.42.0 NDI (94), length 3: 1.3.16 ARCH (93), length 2: 7 Vendor-Class (60), length 32: "PXEClient:Arch:00007:UNDI:003016" 16:08:35.140038 IP (tos 0x0, ttl 64, id 29394, offset 0, flags [none], proto UDP (17), length 375) 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from <client’s mac address>(oui Unknown), length 347, xid 0x92c2c13c, secs 28, Flags [Broadcast] Client-Ethernet-Address <client’s mac address>(oui Unknown) Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message (53), length 1: Discover MSZ (57), length 2: 1472 Parameter-Request (55), length 35: Subnet-Mask (1), Time-Zone (2), Default-Gateway (3), Time-Server (4) IEN-Name-Server (5), Domain-Name-Server (6), Hostname (12), BS (13) Domain-Name (15), RP (17), EP (18), RSZ (22) TTL (23), BR (28), YD (40), YS (41) NTP (42), Vendor-Option (43), Requested-IP (50), Lease-Time (51) Server-ID (54), RN (58), RB (59), Vendor-Class (60) TFTP (66), BF (67), GUID (97), Unknown (128) Unknown (129), Unknown (130), Unknown (131), Unknown (132) Unknown (133), Unknown (134), Unknown (135) GUID (97), length 17: 0.128.226.9.231.152.246.237.17.132.7.227.255.61.233.42.0 NDI (94), length 3: 1.3.16 ARCH (93), length 2: 7 Vendor-Class (60), length 32: "PXEClient:Arch:00007:UNDI:003016" ^C 4 packets captured 4 packets received by filter 0 packets dropped by kernel

In case it is informative, my initial config file yields a few cycles of

cpdump: listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 16:25:14.333731 IP (tos 0x0, ttl 64, id 3165, offset 0, flags [none], proto UDP (17), length 375) 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from <client’s mac address>(oui Unknown), length 347, xid 0xf2335aca, Flags [Broadcast] Client-Ethernet-Address <client’s mac address>(oui Unknown) Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message (53), length 1: Discover MSZ (57), length 2: 1472 Parameter-Request (55), length 35: Subnet-Mask (1), Time-Zone (2), Default-Gateway (3), Time-Server (4) IEN-Name-Server (5), Domain-Name-Server (6), Hostname (12), BS (13) Domain-Name (15), RP (17), EP (18), RSZ (22) TTL (23), BR (28), YD (40), YS (41) NTP (42), Vendor-Option (43), Requested-IP (50), Lease-Time (51) Server-ID (54), RN (58), RB (59), Vendor-Class (60) TFTP (66), BF (67), GUID (97), Unknown (128) Unknown (129), Unknown (130), Unknown (131), Unknown (132) Unknown (133), Unknown (134), Unknown (135) GUID (97), length 17: 0.128.226.9.231.152.246.237.17.132.7.227.255.61.233.42.0 NDI (94), length 3: 1.3.16 ARCH (93), length 2: 7 Vendor-Class (60), length 32: "PXEClient:Arch:00007:UNDI:003016" 16:25:14.334222 IP (tos 0xc0, ttl 64, id 43055, offset 0, flags [none], proto UDP (17), length 328) CREEDfog.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 300, xid 0xf2335aca, Flags [Broadcast] Server-IP CREEDfog Client-Ethernet-Address <client’s mac address>(oui Unknown) Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message (53), length 1: Offer Server-ID (54), length 4: CREEDfog Vendor-Class (60), length 9: "PXEClient" GUID (97), length 17: 0.128.226.9.231.152.246.237.17.132.7.227.255.61.233.42.0

@rpycroft Thank you for the writeup. The elimination you did up front — cables,
link speed, compression, RAID vs AHCI, all three storage nodes, BIOS, iPXE
files — saved me a lot of guessing, and the commit you linked is the right one.

You’re right that it’s the r8169 switch, but not quite in the way it looks.
The in-kernel driver isn’t worse than the vendor one here. It exposed something
we’d had broken for nine years without knowing.

Two kernel options that are on by default upstream have been off in the FOS
kernel config since 2016:

# CONFIG_PCIEASPM is not set # CONFIG_PCI_MMCONFIG is not set

ASPM is PCIe link power management — it lets the link drop into a low power
state when idle and wake when there’s traffic. Waking up is the expensive part.

Nobody noticed for nine years because the Realtek vendor drivers we used to
ship handled ASPM themselves. Our r8168 build had dynamic ASPM compiled in,
which effectively switched ASPM off any time packets were moving. It didn’t
matter that the kernel couldn’t manage it, because the driver never asked the
kernel to.

r8169 does it the normal way: it asks the PCI core to disable L1 once, when the
card is probed. That’s where this bites. With CONFIG_PCIEASPM off, the function
it calls isn’t a function at all — it’s a stub that returns “success” without
doing anything. So r8169 is told the OS disabled L1, marks ASPM as being under
OS control, and then goes on to enable ASPM and L1.2 on the card — on a
kernel with no ASPM code in it that disabled nothing. Not one line in the logs
to say so.

That “falling back to CSI” line you included is the second option. Without
CONFIG_PCI_MMCONFIG the kernel only sees the first 256 bytes of each device’s
config space, and the registers controlling the L1 sub-states live past that.
So even a kernel that wanted to fix this couldn’t reach them. Including that
line is what let me tie the two together, so thank you for pasting it.

That also explains the two things in your report that looked strangest:

Why UEFI and not legacy — ASPM is programmed by the firmware, and Dell’s UEFI
path turns L1 on where the legacy/CSM path leaves it off. Our kernel couldn’t
change it either way, so whatever the firmware picked stuck. Legacy was never
actually faster; you were getting a machine where ASPM was already off.

Why deploys and not captures — a deploy is your client receiving. The link
goes quiet between bursts from the server, drops into L1.2, and pays the wake
cost over and over. A capture is the client sending, which keeps the link
busy so it never gets the chance to sleep. That’s your 15 GB/min upload sitting
next to a 1.2 GB/min download on the same cable.

Worth saying before anyone suggests it: adding pcie_aspm=off to the kernel
arguments does nothing on a FOS kernel. The code registering that argument is
inside the same #ifdef, so it’s compiled out too. It would look like you tried
the standard fix and it didn’t help.

I’ve built you an experimental x64 kernel with both options turned back on:

https://github.com/FOGProject/fos/releases/tag/EXP_20260805-123232 sha256 6fadc7204889bc76fe943520319044940c24654365a523d5df7adf8327a88d25

It’s the kernel only — your init is untouched and doesn’t need changing. Back
up your current /var/www/html/fog/service/ipxe/bzImage, drop this one in its
place, and deploy to one of the 3070s. Putting the old file back is the whole
rollback if it doesn’t help.

If you want to confirm the diagnosis yourself first, it takes about a minute
and doesn’t need my kernel at all. Boot a 3070 to a shell in debug mode:

lspci -nn | grep -i ethernet # note the Realtek address, e.g. 02:00.0 lspci -t # note the root port above it, e.g. 00:1c.5 setpci -s 02:00.0 CAP_EXP+10.w setpci -s 00:1c.5 CAP_EXP+10.w

The bottom two bits of each value are the ASPM setting — 0 is off, 2 is L1,
3 is L0s+L1. Run it once booted UEFI and once booted legacy. If I have this
right, UEFI shows a 2 or a 3 and legacy shows a 0.

And to watch it fix itself, clear both (root port first) and rerun the deploy
without rebooting:

setpci -s 00:1c.5 CAP_EXP+10.w=0000 setpci -s 02:00.0 CAP_EXP+10.w=0000

That’s not a fix you can keep — it’s gone on the next boot — but if the speed
jumps back to 6-10 GB/min, that confirms it before you swap any files.

One loose end: could you post the raw output of ethtool -k <interface>? You
mentioned tx checksumming being off and I want to be sure I’m not waving away a
second, separate problem. It shouldn’t be able to cause a download-only
slowdown, but I’d rather look than assume.

Thanks for the report and for offering to test — I’m taking you up on it.

@Lucas26 Thanks for letting us know and should be fixed in the latest.

Also found a similar class of bug for FTP ports and we are porting the FTP side to dev-branch as we speak.

Please let us know if this is working now?

I have figured out the issue. I was looking in the wrong place in BIOS for the secure boot setting which was on. Turning off secure boot fixed the process which runs perfectly now.

@Alun Thank you and added to documentation and within the installer to try to more cleanly account for this.

@Valer Thanks for the report and the writeup, that’s a good catch. You’re right about the cause. Debian 13 dropped the sysv-rc-conf package since it’s fully on systemd now, so the installer chokes trying to apt-get it. We never actually use that tool on a systemd system anyway, it only ever got called on the old non-systemd paths, so it was just dead weight on Trixie.

I pushed a fix to working-1.6. It leaves the package off the list on Debian 13 and up, and it also strips it out of a cached .fogsettings so an in-place upgrade doesn’t drag it back in. Older Debian keeps installing it like before so nothing changes there.

If you want to get moving before you pull the update, you can either grab the latest working-1.6 and rerun the installer, or just open /opt/fog/.fogsettings and delete sysv-rc-conf from the packages line, then run the installer again. Either one gets you past it.

Let me know if you hit anything else on Trixie.

Hi everyone, / @george1421

I’ve been trying to capture the SD card of a Raspberry Pi 4 and I’ve hit a wall. I am booting via a USB stick using the EDK2 UEFI firmware, and I have explicitly set the System Table Selection to <Devicetree> in the UEFI settings.

I managed to get past the iPXE phase without issues, and the capture task successfully loads the FOS kernel and init. However, the boot process stops right after with the following error (see attached image):
No network interfaces found, your kernel is most probably missing the correct driver!

Current Setup & Debug info:

FOG version: 1.5.10.2149

Hardware: Raspberry Pi 4 (trying to capture the built-in SD card)

Boot method: USB stick with UEFI firmware

Kernel version: 6.18.38

Init version: 20260726

(No PCI network hardware detected - obviously, as it’s the SoC’s built-in NIC)

I think it seems the current experimental ARM kernel is missing the built-in Pi 4 network driver (bcmgenet). Unfortunately, using an external USB-to-LAN adapter is not an option for me, as the UEFI firmware refuses to boot from it in my environment.

My question is:
Given that I am booting from a USB stick and making it exactly to this point in the capture task, is there any chance or workaround to successfully capture the Pi 4’s SD card? Is there a custom/specific kernel available that includes the built-in Pi network driver, or any other method I can use to bypass this missing driver issue?

Any help or guidance would be greatly appreciated!
20260803_152648.jpg