Slow restoration of Windows 11 with FOG on Proxmox

rurap

@george1421
I checked restoring Windows 10 and it also restores slowly. But that already follows from what you wrote earlier, I changed various settings in BIOS (disabled C-state for the processor) and still nothing.

Write speeds on the disk are higher than my old SSDs on SATA, unless NVMe is somehow "messing up. Maybe I will install the system on a SATA SSD.
I’m considering connecting a new network card and testing the restoration on it.

I’m not sure what you meant by the FOS kernel, Fog is running on the latest version 1.5.10.1629 on Ubuntu 24.04.1, the system kernel is 6.8 so if that’s what you meant, then it’s probably the latest one."

george1421

@rurap said in Slow restoration of Windows 11 with FOG on Proxmox:

I’m not sure what you meant by the FOS kernel

There is a customized linux distro that gets copied to the target computer during pxe booting. You will see that as bzImage and init.xz. To check the version of FOS linux from within the web ui go to fog settings->kernel update. From there it will tell you the version and give you options to update it if needed.

So from your testing you can rule out the OS being deployed. I kind if figured that but ruling it out is always good at helping us narrow down the root of the issue.

So now its down to the network adapter and or the disk controller/nvme. If we could use your idea of process of elimination (one you verify you are on the latest version of FOS Linux) to see where the problem isn’t.

Also we might want to schedule a debug deployment so we can get to the command line on the target system within FOS linux. I would search /var/log/syslog on the target computer for the keyword firmware to see if there are any error messages wanting a specific version of a firmware driver.

rurap

@george1421
In the place you indicated, I see different kernels, but I don’t know where the version of this kernel is.

george1421

@rurap Well, I thought it said the current version at the top…

OK for a plan B then, with the fog server console, navigate to /var/www/html/fog/service/ipxe directory. In that directory run this command file bzImage. The file command should tell you about that kernel image with a version number embedded in it. Note the version number and post it here.

rurap

@george1421

The result of the command I received:

bzImage: Linux kernel x86 boot executable bzImage, version 6.6.49 (runner@fv-az1756-740) #1 SMP PREEMPT_DYNAMIC Thu Sep 5 00:21:28 UTC 2024, RO-rootFS, swap_dev 0X9, Normal VGA

rurap

@george1421

I think I have found the problem - the network card. I installed another network card and it turned out to be practically the same card as the built-in one, and the restoration speed did not increase. I had another one that was recognized in the system as TP-link instead of Realtek, and after replacing it, the restoration speed suddenly reached 17.33GB/min. Can someone now explain to me what’s going on with this Realtek? The integrated card is a Realtek RTL8821CE. I also have an M.2 WiFi card with Bluetooth in a Dell Vostro PC, an Intel AX201, but it will likely be difficult to use it for booting and restoring computers. Any ideas?

Below are the differences between the mentioned network cards I installed in the Vostro.

george1421

@rurap From historical experience if we have an issue with a network card/chip set it will be realtek.

For that realtek nic, can you get the hardware ID of that card, in windows you can get it from the device manager vendor and Id numbers. They will be 4 hex digit codes both the vendor and device id.

You can get them from FOS linux running on the target computer. It will probably be easiest to get from FOS Linux since I will need you to get some info from there for the second part of the question.

Schedule a debug capture/deploy to one of these vostros. Before you hit the scheduled task button tick the debug checkbox. Then schedule the task.

PXE boot the target computer, after several screens of text you need to clear with the enter key you will be dropped to the command prompt.

Key in the following
lspci -nn | grep -i net
Search for the Realtek nic in question and capture the hex codes in the square brackets in the form of [XXXX:XXXX]

Some nics require special firmware to work correctly with linux. Run this command to see if any firmware messages were logged. I should know this by now but I forget if the log is syslog or messages file. So you might have to adjust.

grep -i -e firm /var/log/syslog

Hint: If you want to make it easier for copying and pasting to the target computer do the following after you pxe boot into debug mode.

Get the IP address of the target computer with the following command ip a s
Change root’s password with passwd make it something simple like hello it will be reset at the next reboot.
Now using putty from a windows computer or ssh from a linux computer connect to the target system using the above two bits of info. Login as root and the password you just set.

From there you can copy and paste using the apps.

rurap

@george1421

Information about the installed cards:

00:14.3 Network controller [0280]: Intel Corporation Alder Lake-S PCH CNVi WiFi [8086:7af0] (rev 11)
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8161] (rev 15)
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) - This is the built-in network card

You can see a total of 3 network cards, as I mentioned, one is an Intel WiFi card and the other two are Realtek cards, which appear to be identical."

grep -i -e firm /var/log/messages - this command did not return any information.

george1421

@rurap said in Slow restoration of Windows 11 with FOG on Proxmox:

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8161] (rev 15)
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) - This is the built-in network card

and the other two are Realtek cards, which appear to be identical.

Technically they are not identical, they have the same family name but one is a 8161 and the other is a 8168 of the rev 15 generation.

Just confirming that the /var/log/messages file exists and the /var/log/syslog one doesn’t ? The 8168 model has been around for years as you can see by the rev numbers. I believe they need nic specific firmware to operate. That should be called out in the boot up messages.

In researching this it seems one instant the 8169 driver was being installed instead of the 8168 linux kernel driver. This will take some looking by lspci -knn | more will list out all installed pci hardware with the “kernel drivers in use” for the hardware. As a hint for looking through the big list the section you are interested in starts with “03:00.0 Ethernet controller” since that is the built in nic you found in the previous post. Lets see if its using the 8169 driver instead.

Edit: I keep seeing reference to these kernel parameters in posts about debugging this nic r8168.aspm=0 r8168.eee_enable=0 pcie_aspm=off Not sure the importance, but noting it here for future reference.

Edit2: This command may help give the answer on driver than looking through the entire list of lspci commands: inxi -Naz

rurap

@george1421 said in Slow restoration of Windows 11 with FOG on Proxmox:

Just confirming that the /var/log/messages file exists and the /var/log/syslog one doesn’t ?

I only have two files there, nothing more: resolv.conf and messages

@george1421 said in Slow restoration of Windows 11 with FOG on Proxmox:

In researching this it seems one instant the 8169 driver was being installed instead of the 8168 linux kernel driver. This will take some looking by lspci -knn | more will list out all installed pci hardware with the “kernel drivers in use” for the hardware. As a hint for looking through the big list the section you are interested in starts with “03:00.0 Ethernet controller” since that is the built in nic you found in the previous post. Lets see if its using the 8169 driver instead.

I removed the network card from the PCI slot, and only the built-in one remains, so the section probably starts at [0200] and it looks like you are right.:

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1028:0acf]
Kernel driver in use: r8169

If I understood correctly, the problem is due to the wrong network card driver? What to do now, how to install the correct driver?

@george1421 said in Slow restoration of Windows 11 with FOG on Proxmox:

This command may help give the answer on driver than looking through the entire list of lspci commands: inxi -Naz

Unfortunately, I don’t have this command in the command line, but it seems it’s not needed since I’ve already got the answer

rurap

@rurap
OK, I tried it myself but failed. Somehow I couldn’t manage to attach the drivers to the system kernel, I don’t even know if I’m doing it right. I tried to follow this documentation:

https://docs.fogproject.org/en/latest/kb/reference/compile-fos-kernel/#arm-64-bit

I downloaded the drivers from here:

https://packages.ubuntu.com/noble/r8168-dkms.

I don’t know where to install these drivers, I don’t even know which files. I’m just groping in the dark and still don’t know how to do it, or if it’s even possible. Can anyone explain how to do it?

rurap

Hello again, after some time, I still haven’t been able to solve the problem. I started looking for different solutions, I installed the latest version of FOG, but so far nothing has helped. In the system logs in the file /var/log/messages, I found this line:

code_text
Feb 11 11:48:52 fogclient kern.noticekernel: r8169 0000:03:00.0 enp3s0: No native access to PCI extended config space, falling back to CSI

Since the driver cannot directly access the PCI and falls back to CSI, is this an issue with the driver itself, or could it be something else?

I have attached the entire log file in the post. Could someone take a look at this?

messages.txt

rurap

Hello again, I conducted additional tests. In debug mode on the client, I ran network tests between the client and the fog server using the iperf server. The results do not indicate any network issues:

iperf -c 192.168.25.11 
------------------------------------------------------------ 
Client connecting to 192.168.25.11, TCP port 5001 TCP 
window size: 16.0 KByte (default) 
------------------------------------------------------------ [ 1] 
local 192.168.25.49 port 35204 connected with 192.168.25.11 port 5001 
(icwnd/mss/irtt= 14/1448/924) 
[ ID] Interval Transfer Bandwidth [ 1] 0.00-10.02 sec 1.09 GBytes 938 Mbits/sec

I also conducted disk write tests, and they also do not indicate any issues with the disk:

bash
dd if=/dev/zero of=/tmp/testfile bs=1M count=1000 status=progress
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.17635 s, 5.9 GB/s

I also created an uncompressed image, and it also restored slowly, with transfer speeds below 1GB/s, rather around 500-600 MB/s, which results in even lower speeds.

I changed the PXE environment boot file, set it to realtek.efi, and then changed it to snp.efi.The snp.efi visibly improved the FOS loading process, but the image restoration is still slow. Can the FOS system use a different driver during restoration than it does when operating in debug mode?

Slow restoration of Windows 11 with FOG on Proxmox

194

12.0k

17.3k

155.2k