Slow restoration of Windows 11 with FOG on Proxmox

rurap

Good morning, I am using FOG on Proxmox. There are no problems with imaging and restoring Windows 10. The speed of restoring 16 computers with Windows 10 via multicast ranges between 10-13GB/min. A single computer is 14-15GB/min. The problem arises with Windows 11. The same FOG, the same settings, supposedly everything is the same, but the restoration is happening at a speed of 700MB/min - 1.4GB/min and takes far too long.

I restore Windows 10 on different hardware: Dell Optiplex 7010, these computers are to be replaced by Dell Vostro.

Image settings for Windows 11(the same as for Windows 10) :

Operating System: Windows 10,
Image Type: Multiple Partition Image - Single Disk (not Resizable),
Image Manager tested Partclone Gzip and Zstd. Zstd creates the image twice as fast but restores at the same speed.
The computer I want to restore to is a Dell Vostro 3910 with an i12400 and a 256GB KIOXIA NVMe drive.

Windows 10

I don’t know where to start. is it more likely a hardware problem (Vostro 3910 - maybe BIOS settings, disk too slow for writing?), Proxmox (VM settings?) or FOG? No errors, the system works after restoration.

george1421

@rurap OK just to be clear. Your fog server is running on proxmox. You are only restoring to physical computers. The physical computers in question are Dell 7010 (about 10 years old) and Dell Vostro 3910 with a 12 Gen processor writing to a nvme drive.

Windows 10 deploying to both systems are > 6.5GB/min according to partclone
Windows 11 deploying to both systems are < 2GB/min according to partclone

Do I understand the situation correctly?

If this is the case we can probably rule out the fog server and proxmox as a root cause. FOG on a capture or deploy are really not a factor in imaging because at this point its only moving data blocks between the network interface and local nonvolatile storage. All of the heavy lifting is being done by the target computer.

Also, what version of FOG are you running as well as what version of the FOS Linux (fog ui->fog settings->kernel update) are you running. Make sure you are running the latest 6.6.x version. Having an old version of FOG and FOS Linux kernel may cause this type of issue.

rurap

As for Windows 10, I am restoring it only on Dell 7010 computers, tomorrow I planned to test this system on a new one. As for the FOG version, it’s the latest one, I will check FOS Linux tomorrow. I thought similarly that It’s neither FOG nor Proxmox., but I have no idea how to check the target computer.

Ethernet on Vostro is - Realtek RTL8111HSD - Could the network card have its transfer speed limited during image restoration? I will check the network card settings in BIOS tomorrow.

george1421

@rurap Just to add an additional bit of information the partclone “speed” is a composite rating of

Speed the fog server can move the image from nv storage to the network interface.
Transmission speed over the network. (these first two variables are probably not changing between win10 and win11 deployments.
Target system ingest of image from network adapter to main memory
Decompression of image (using zstd compressor [regardless of compressor used])
Writing the decompressed image to disk.

If we had to rule out differences between Dell 7010 and 3910 the decompression of the image is probably not a factor because of the speed differences between a 4th Gen processor and a 12th Gen Intel processor (the Pcore-Ecore might be an issue depending on the FOS kernel). Looking at the truth table I would say the issue is either with #3 or #5.

rurap

@george1421
I checked restoring Windows 10 and it also restores slowly. But that already follows from what you wrote earlier, I changed various settings in BIOS (disabled C-state for the processor) and still nothing.

Write speeds on the disk are higher than my old SSDs on SATA, unless NVMe is somehow "messing up. Maybe I will install the system on a SATA SSD.
I’m considering connecting a new network card and testing the restoration on it.

I’m not sure what you meant by the FOS kernel, Fog is running on the latest version 1.5.10.1629 on Ubuntu 24.04.1, the system kernel is 6.8 so if that’s what you meant, then it’s probably the latest one."

george1421

@rurap said in Slow restoration of Windows 11 with FOG on Proxmox:

I’m not sure what you meant by the FOS kernel

There is a customized linux distro that gets copied to the target computer during pxe booting. You will see that as bzImage and init.xz. To check the version of FOS linux from within the web ui go to fog settings->kernel update. From there it will tell you the version and give you options to update it if needed.

So from your testing you can rule out the OS being deployed. I kind if figured that but ruling it out is always good at helping us narrow down the root of the issue.

So now its down to the network adapter and or the disk controller/nvme. If we could use your idea of process of elimination (one you verify you are on the latest version of FOS Linux) to see where the problem isn’t.

Also we might want to schedule a debug deployment so we can get to the command line on the target system within FOS linux. I would search /var/log/syslog on the target computer for the keyword firmware to see if there are any error messages wanting a specific version of a firmware driver.

rurap

@george1421
In the place you indicated, I see different kernels, but I don’t know where the version of this kernel is.

george1421

@rurap Well, I thought it said the current version at the top…

OK for a plan B then, with the fog server console, navigate to /var/www/html/fog/service/ipxe directory. In that directory run this command file bzImage. The file command should tell you about that kernel image with a version number embedded in it. Note the version number and post it here.

rurap

@george1421

The result of the command I received:

bzImage: Linux kernel x86 boot executable bzImage, version 6.6.49 (runner@fv-az1756-740) #1 SMP PREEMPT_DYNAMIC Thu Sep 5 00:21:28 UTC 2024, RO-rootFS, swap_dev 0X9, Normal VGA

rurap

@george1421

I think I have found the problem - the network card. I installed another network card and it turned out to be practically the same card as the built-in one, and the restoration speed did not increase. I had another one that was recognized in the system as TP-link instead of Realtek, and after replacing it, the restoration speed suddenly reached 17.33GB/min. Can someone now explain to me what’s going on with this Realtek? The integrated card is a Realtek RTL8821CE. I also have an M.2 WiFi card with Bluetooth in a Dell Vostro PC, an Intel AX201, but it will likely be difficult to use it for booting and restoring computers. Any ideas?

Below are the differences between the mentioned network cards I installed in the Vostro.

george1421

@rurap From historical experience if we have an issue with a network card/chip set it will be realtek.

For that realtek nic, can you get the hardware ID of that card, in windows you can get it from the device manager vendor and Id numbers. They will be 4 hex digit codes both the vendor and device id.

You can get them from FOS linux running on the target computer. It will probably be easiest to get from FOS Linux since I will need you to get some info from there for the second part of the question.

Schedule a debug capture/deploy to one of these vostros. Before you hit the scheduled task button tick the debug checkbox. Then schedule the task.

PXE boot the target computer, after several screens of text you need to clear with the enter key you will be dropped to the command prompt.

Key in the following
lspci -nn | grep -i net
Search for the Realtek nic in question and capture the hex codes in the square brackets in the form of [XXXX:XXXX]

Some nics require special firmware to work correctly with linux. Run this command to see if any firmware messages were logged. I should know this by now but I forget if the log is syslog or messages file. So you might have to adjust.

grep -i -e firm /var/log/syslog

Hint: If you want to make it easier for copying and pasting to the target computer do the following after you pxe boot into debug mode.

Get the IP address of the target computer with the following command ip a s
Change root’s password with passwd make it something simple like hello it will be reset at the next reboot.
Now using putty from a windows computer or ssh from a linux computer connect to the target system using the above two bits of info. Login as root and the password you just set.

From there you can copy and paste using the apps.

rurap

@george1421

Information about the installed cards:

00:14.3 Network controller [0280]: Intel Corporation Alder Lake-S PCH CNVi WiFi [8086:7af0] (rev 11)
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8161] (rev 15)
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) - This is the built-in network card

You can see a total of 3 network cards, as I mentioned, one is an Intel WiFi card and the other two are Realtek cards, which appear to be identical."

grep -i -e firm /var/log/messages - this command did not return any information.

george1421

@rurap said in Slow restoration of Windows 11 with FOG on Proxmox:

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8161] (rev 15)
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) - This is the built-in network card

and the other two are Realtek cards, which appear to be identical.

Technically they are not identical, they have the same family name but one is a 8161 and the other is a 8168 of the rev 15 generation.

Just confirming that the /var/log/messages file exists and the /var/log/syslog one doesn’t ? The 8168 model has been around for years as you can see by the rev numbers. I believe they need nic specific firmware to operate. That should be called out in the boot up messages.

In researching this it seems one instant the 8169 driver was being installed instead of the 8168 linux kernel driver. This will take some looking by lspci -knn | more will list out all installed pci hardware with the “kernel drivers in use” for the hardware. As a hint for looking through the big list the section you are interested in starts with “03:00.0 Ethernet controller” since that is the built in nic you found in the previous post. Lets see if its using the 8169 driver instead.

Edit: I keep seeing reference to these kernel parameters in posts about debugging this nic r8168.aspm=0 r8168.eee_enable=0 pcie_aspm=off Not sure the importance, but noting it here for future reference.

Edit2: This command may help give the answer on driver than looking through the entire list of lspci commands: inxi -Naz

rurap

@george1421 said in Slow restoration of Windows 11 with FOG on Proxmox:

Just confirming that the /var/log/messages file exists and the /var/log/syslog one doesn’t ?

I only have two files there, nothing more: resolv.conf and messages

@george1421 said in Slow restoration of Windows 11 with FOG on Proxmox:

In researching this it seems one instant the 8169 driver was being installed instead of the 8168 linux kernel driver. This will take some looking by lspci -knn | more will list out all installed pci hardware with the “kernel drivers in use” for the hardware. As a hint for looking through the big list the section you are interested in starts with “03:00.0 Ethernet controller” since that is the built in nic you found in the previous post. Lets see if its using the 8169 driver instead.

I removed the network card from the PCI slot, and only the built-in one remains, so the section probably starts at [0200] and it looks like you are right.:

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1028:0acf]
Kernel driver in use: r8169

If I understood correctly, the problem is due to the wrong network card driver? What to do now, how to install the correct driver?

@george1421 said in Slow restoration of Windows 11 with FOG on Proxmox:

This command may help give the answer on driver than looking through the entire list of lspci commands: inxi -Naz

Unfortunately, I don’t have this command in the command line, but it seems it’s not needed since I’ve already got the answer

Slow restoration of Windows 11 with FOG on Proxmox

141

11.7k

17.2k

154.7k