I’m currently testing init.xz images built by myself using the partclone-0320 branch. In addition, these images have, for my part, added these two commits of mine from github that fix bugs with BTRFS (https://github.com/FOGProject/fos/pull/47 https://github.com/FOGProject/fos/pull/45). The whole thing was built using Buildroot 2022.02.5, which fixes bugs related to udev (https://github.com/FOGProject/fos/issues/46). I know it’s too many changes to treat my experience as relevant to the addition of this particular partclone version, but I think it’s worth sharing anyway.

The FOG server on which this custom init.xz runs is based on Fedora 36 (/images on XFS), the latest (at the time of writing this) development version of FOG, compiled kernel 5.15.71, and an updated version of UDPcast to 20200328 (the latest does not work with FOG). This server, runs in production, so there are 30 images restored each day, and sometimes more (mostly Windows 10 and Ubuntu). But it happens to image other computers that have other systems, such as Fedora 36 Workstation with the BTRFS file system. I mainly use Multicast restore (computers in the school computer lab), but sometimes I need to restore one computer and I use Unicast.

The previous FOG server was based on Debian 10 (/images on EXT4), FOG 1.5.9 and the latest official kernel and init.xz. The situation has changed dramatically after the migration.

On the old system, using Unicast I was getting speeds of around 7-8GB/min (for both restore and capture). With Multicast, this speed increased to 12GB/min.

After I built the server on Fedora, I used init.xz from the partclone-0320 branch without patches and the official 5.15.68 kernel and the speed increased to 14GB/min using both Multicast and Unicast. The capture speed went up to 9GB/min. After changing init.xz to the current one, the speeds have not changed, but at least I can safely restore the BTRFS file system without any errors.

I don’t know how much of this is due to Partclone 3.20 and how much is due to migrating the system to Fedora with XFS, but I can say that so far Partclone 3.20 is running very stably and hasn’t crashed yet. And I have already restored images based on NTFS, XFS, EXT4 and BTRFS. If I only notice any flaws with the operation of the whole thing I will describe them. But so far I have no complaints about my system. If it continues to perform as well as it does now I will migrate the FOGs in other computer labs to what I am using in this particular one.

@george1421

Now with a browser go to http://<fog_server_ip>/fog/service/ipxe/boot.php?mac=00:00:00:00:00:00 That will display the text behind the iPXE menu. On that page search for default. That will be the section that is called when the timeout happens and boots from the local hard drive. By changing globally the exit modes for both bios and uefi to exit, it should put the exit command in the iPXE Menu script.

And it is putting the exit command in the iPXE Menu script

choose --default fog.local --timeout 5000 target && goto ${target}
:fog.local
exit || goto MENU

I’ve tried to mess with the embedded script itself and I found that if I replace

:netboot
chain tftp://${next-server}/default.ipxe ||
prompt --key s --timeout 10000 Chainloading failed, hit 's' for the iPXE shell; reboot in 10 seconds && shell || reboot

with this

chain tftp://${next-server}/default.ipxe || echo Chainloading failed

and try to “Boot from Hard Drive” using “EXIT” method, it is WORKING! The “Chainloading failed” is not echoing back to me. But if I write this like that

chain tftp://${next-server}/default.ipxe || 
echo Chainloading failed

the “Chainloading failed” is echoing back to me. So I guess the issue here is not with the chain command but with the syntax.

Apparently

command || command

is not the same as

command ||
command

So I’ve tried to leave this “prompt” command but in this manner:

:chainloadfailed
prompt --key s --timeout 10000 Chainloading failed, hit 's' for the iPXE shell; reboot in 10 seconds && shell || reboot

:netboot
chain tftp://${next-server}/default.ipxe || goto chainloadfailed

And now it works! When I use EXIT by clicking “Boot from Hard Drive”, iPXE is correctly exiting. And if I rename default.ipxe on my server to something else (to simulate failed chainloading), the “Chainloading failed, hit ‘s’ (…)” message is appering so I guess that the core of this issue is incorrect syntax in iPXE script, and the solution is to write this like I did.

Straight away I say that if I write it like this (in one line)

:netboot
chain tftp://${next-server}/default.ipxe || prompt --key s --timeout 10000 Chainloading failed, hit 's' for the iPXE shell; reboot in 10 seconds && shell || reboot

the script is not working, beacuse it is dropping me straight to iPXE shell, so the better solution is to write this like this:

:chainloadfailed
prompt --key s --timeout 10000 Chainloading failed, hit 's' for the iPXE shell; reboot in 10 seconds && shell || reboot

:netboot
chain tftp://${next-server}/default.ipxe || goto chainloadfailed

So I guess, we’ve solved this mistery. Next-server variable is working - I’ve tried to echo it and it echoed IP address of my FOG server, by the way.

@sebastian-roth
As it turns out, this topic is a bit complicated.

The “r8169” driver built into the kernel supports the following Realtek chips:
RTL8169 Gigabit Ethernet
RTL8168 Gigabit Ethernet
RTL8101 Fast Ethernet
RTL8125 2.5GBit Ethernet

Whereas the “r8168” driver provided by Realtek only supports:
RTL8168
RTL8111

And installing r8168 REQUIRES removing r8169 from the kernel. Therefore, in order to maintain functionality, I was forced to integrate into the kernel not only r8168 from Realtek, but also other drivers provided by Realtek. These drivers are:
r8101 - support for RTL8101 and related chipsets
r8125 - support for RTL8125 and related chipsets
r8169 - support for RTL8169 and related chipsets (DO NOT MISS THE KERNEL DRIVER!!! These are not the same drivers)

Only by integrating these four drivers can the integrated r8169 driver be replaced while maintaining support for all chips. And removing the integrated r8169 driver is required - as I mentioned.

I can provide a patch file that I use to compile modified kernels (the patch completely swaps the integrated r8169 for these 5 drivers mentioned earlier).

Versions I use:
r8101 - 1.037.01
r8125 - 9.010.01
r8168 - 8.050.03
r8169 - 6.031.00 (recall, this is also the name of the Realtek driver, which currently has nothing to do with the one built into the kernel)

There is one more important thing!
The Realtek drivers r8101 and r8168 have a function called “mdio_real_read” in the r8101_n.c and r8168_n.c files. This function is not set as “static” so the linker throws an error if you integrate these two drivers in the kernel. You need to make small modifications and add “static” in both files to these functions (r8125 already has static added). My patch already contains this modification.

Here you are right. Partclone shows the speed of writing to the disk, not the speed of downloading the image from the server. I wouldn’t look for the increase in write speed in improving the network parameters either, but more in the storage parameters of my machine. I switched /images to XFS and on top of that I made sure that in Fedora, no unnecessary background services were running. I also forgot to add that I tweaked the parameters of the machine itself - from 6 cores to 12 and from 8GB of RAM to 12GB. Previously, the machine may simply not have been able to keep up with writes and reads from the array.

I also don’t rule out that changes to the kernel and init also had an impact on the speed increase. It is likely that the SPECULATION MITIGATIONS option may have had some bearing on the matter. The iperf tests showed, full 1Gbps speed from the FOG server to the FOS, both with the old setup and the new one. So by saying that the server setup matters, I’m referring specifically to the server’s hardware configuration and disk configuration. I’ve made too many changes to the FOG to say unequivocally what specifically caused such an increase in speed - I’m no less pleased that I was able to squeeze out something more. I am happy with what I have and am unlikely to try to get more. The important thing for me is that the current speeds are stable and nothing has crashed yet.

Of course - the more variety the better the test results will be. I will slowly start to deploy my kernels and inits in other computer labs at school, where there is also diversity in computers and configurations, so I will know from the lab supervisors if something on their hardware does not work.

Funny thing, because there are still some issues with each lab having a different hardware configuration, but at least now it’s useful for FOG testing.

@george1421 said in CentOS Partition Count Failed:

@wt_101 Would you switch to the dev branch to upgrade to 1.5.9.200 or later. The reason is that version of partclone has been upgraded to 0.3.20 from 0.3.13.

In theory you should be able to use single disk resizable with XFS file system. I have not personally tried cloning an XFS system but as long as its a standard partition (not lvm) it should just work.

I also tried cloning an XFS partition and it doesn’t work on the “Resizable” setting either. I think it is related to this code in the fog.upload file of the FOS system.

            if [[ $ntfscnt -eq 0 && $extfscnt -eq 0 && $btrfscnt -eq 0 ]]; then
                echo "Failed"
                debugPause
                handleError "No resizable partitions found ($0)\n   Args Passed: $*"
            fi

The script checks how many NTFS, EXT and BTRFS partitions there are and if the number is zero, it throws an error. For systems that only have XFS partitions, this causes the above error as the script doesn’t count how many XFS partitions there are (because XFS partitions can’t be resized anyway).

@sebastian-roth

It’s a good idea to release a separate kernel build, with integrated Realtek drivers. At the moment I have to deal with FOG almost every day and I would be able to see to it that the patch remains compatible (but I don’t know what the future will bring). People could test at their place how this custom kernel works and indeed as if there would be no problems, it could be used as default kernel. Of course, if the ‘r8169’ driver built into the kernel gets fixed, then we could go back to generic kernel builds.

Hmmmmm, I might have just stumbled upon a pain licence issue here. The Realtak drivers are released under GPLv2-only

I’m no legal expert, but it seems to me that as long as the kernel is not an integral part of the FOG code (and is only downloaded during installation), it doesn’t matter what license the FOG is under. I just don’t know what it would be like to share pre-built kernels on Github (just sharing a patch would not likely be a problem - the code is public, after all). Nevertheless, it would be better to consult someone knowledgeable about the topic.

Here is link to the patch: https://mega.nz/file/z5swlQjS#RuS5gGVD2Kc2IpQr3YTgEnw8xg3o8vaABeDBwuDEST0

@sebastian-roth
As it turns out, this topic is a bit complicated.

The “r8169” driver built into the kernel supports the following Realtek chips:
RTL8169 Gigabit Ethernet
RTL8168 Gigabit Ethernet
RTL8101 Fast Ethernet
RTL8125 2.5GBit Ethernet

Whereas the “r8168” driver provided by Realtek only supports:
RTL8168
RTL8111

And installing r8168 REQUIRES removing r8169 from the kernel. Therefore, in order to maintain functionality, I was forced to integrate into the kernel not only r8168 from Realtek, but also other drivers provided by Realtek. These drivers are:
r8101 - support for RTL8101 and related chipsets
r8125 - support for RTL8125 and related chipsets
r8169 - support for RTL8169 and related chipsets (DO NOT MISS THE KERNEL DRIVER!!! These are not the same drivers)

Only by integrating these four drivers can the integrated r8169 driver be replaced while maintaining support for all chips. And removing the integrated r8169 driver is required - as I mentioned.

I can provide a patch file that I use to compile modified kernels (the patch completely swaps the integrated r8169 for these 5 drivers mentioned earlier).

Versions I use:
r8101 - 1.037.01
r8125 - 9.010.01
r8168 - 8.050.03
r8169 - 6.031.00 (recall, this is also the name of the Realtek driver, which currently has nothing to do with the one built into the kernel)

There is one more important thing!
The Realtek drivers r8101 and r8168 have a function called “mdio_real_read” in the r8101_n.c and r8168_n.c files. This function is not set as “static” so the linker throws an error if you integrate these two drivers in the kernel. You need to make small modifications and add “static” in both files to these functions (r8125 already has static added). My patch already contains this modification.

@george1421
It is likely that the settings for MagicPacket WoL are stored in some kind of memory on the NIC. The same is true on Windows, where there is a “Wake on Magic Packet” option in the Device Manager and its setting is required for WoL to work - in Linux this is done with the command ethtool -s enp1s0 wol g.

Unfortunately, the built-in driver in the kernel - r8169 probably has poorly implemented WoL support, as regardless of the setting with the ethtool tool WoL does not work, and after a reboot the setting returns to Disabled. Browsing through internet resources, I came across information that this is a common bug with some Realtek cards and apparently the ones built into my motherboards suffer from this. The forums recommend installing the r8168 driver from the Realtek website, which does not have this bug. On distributions like Ubuntu this is not a problem - worse with FOS.

So in the end, I replaced the r8169 driver with r8168 in the FOS kernel and created a corresponding patch, which I put in the “patch” directory to make it easier to build my own modified kernels. The problem disappeared.

Additionally, in other computer labs, there are Dell Vostro 3681 PCs which also have a Realtek card and which suffer from an even worse issue - images restore at 3GB/min using Unicast. When I use my modified kernel with the r8168 driver, the speed is 13GB/min. The same situation occurs with the laptops we use - Vostro 3500. Original kernel: 3GB/min, modified kernel: 13GB/min.

I don’t know where the difference comes from, but this driver embedded in the kernel is not very famous anyway - it creates a lot of problems with new models of Realtek cards. I will use the modified kernel for now, until r8169 is fixed.

Hi,
I have a computer lab consisting of 17 computers that are equipped with a built-in Realtek card. I have tried to run WoL on it to restore operating systems faster (sometimes Windows, sometimes Ubuntu) and there is no problem with this under Windows, however, there was a problem under Linux. Well, the r8169 driver embedded in the kernel, does not support WoL correctly. And this is not a problem because all I had to do was install the r8168-dkms package. Unfortunately, FOS, which uses the kernel driver, keeps disabling WoL for me so that I can’t easily boot computers over the network.

Normally under Windows the system activates WoL itself so I only have to start it once and WoL comes back. Linux does not do this and even with the r8168-dkms package I have to use ethtool to activate WoL because after any operation in FOS, WoL is deactivated by the r8169 driver.

Is there any solution to this problem? As a last resort I could build my own kernel with the r8168 driver, but I would like to avoid this.

Of course - the more variety the better the test results will be. I will slowly start to deploy my kernels and inits in other computer labs at school, where there is also diversity in computers and configurations, so I will know from the lab supervisors if something on their hardware does not work.

Funny thing, because there are still some issues with each lab having a different hardware configuration, but at least now it’s useful for FOG testing.

There is no need to rush. The most important thing is that everything works stably - I will keep testing and monitoring my setup.

As for this partclone 0.3.20. The images I am operating on were still created by version 0.3.13 and there is no problem with that. The new ones also work. Here I can agree that APFS is somehow not a popular filesystem, but when it comes to BTRFS, I notice that more and more distributions are pushing this as the main filesystem, e.g. Fedora already offers BTRFS at the start. It may not be as widely used at the moment but it could be soon. “Soon” meaning not tomorrow, so there is still plenty of time to test partclone 0.3.20 before releasing it into production.

Regarding SPECULATION_MITIGATIONS, I’ve compiled a kernel with this option enabled and will test it a bit on different hardware platforms and see if it actually affects this performance that much or not. If not, I’ll leave it enabled - it’s always some sort of security layer, and also disabling this option displays a warning message when FOS starts up, and that doesn’t look too good.

Here you are right. Partclone shows the speed of writing to the disk, not the speed of downloading the image from the server. I wouldn’t look for the increase in write speed in improving the network parameters either, but more in the storage parameters of my machine. I switched /images to XFS and on top of that I made sure that in Fedora, no unnecessary background services were running. I also forgot to add that I tweaked the parameters of the machine itself - from 6 cores to 12 and from 8GB of RAM to 12GB. Previously, the machine may simply not have been able to keep up with writes and reads from the array.

I also don’t rule out that changes to the kernel and init also had an impact on the speed increase. It is likely that the SPECULATION MITIGATIONS option may have had some bearing on the matter. The iperf tests showed, full 1Gbps speed from the FOG server to the FOS, both with the old setup and the new one. So by saying that the server setup matters, I’m referring specifically to the server’s hardware configuration and disk configuration. I’ve made too many changes to the FOG to say unequivocally what specifically caused such an increase in speed - I’m no less pleased that I was able to squeeze out something more. I am happy with what I have and am unlikely to try to get more. The important thing for me is that the current speeds are stable and nothing has crashed yet.

It seems to me that when it comes to increasing the performance of FOG, server configuration makes a big difference.

The previous server was based on Debian 10 with FOG 1.5.9, and the /images directory was on the EXT4 partition. The server was virtualised with VMware ESXi and files were kept on an array of HP 7200RPM SAS drives. The configuration was the default - as it was after the FOG installation. That’s when I was achieving 7-8GB/min on PCs (with NVMe drives) for capture and deployment and 12GB/min for multicast. We are talking about a Windows 10 image (NTFS). With an image with Ubuntu 20.04 (EXT4), the speeds were definitely higher - 18GB/min with Multicast and 12GB/min with Unicast.

However, I wondered if it would be possible to squeeze more out of the whole thing, so I set up a virtual machine based on Fedora 36 with the XFS file system (which supposedly handles large files well, which disk images certainly are). I installed the latest development version of FOG. After installation, I manually compiled Udpcast version 28.03.2020 (the latest one doesn’t work with FOG - it crashes as soon as a multicast session starts). I added the following options to the sysctl.conf file:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.core.rmem_default = 312144
net.core.rmem_max = 312144
net.core.wmem_default = 312144
net.core.wmem_max = 312144
to disable IPv6 (which I do not use) and to increase the socket buffer size. Additionally, I set 128 threads in the NFS settings, as the default is a small number (and the server is running on a vSphere cluster). Still in the FOG itself in the Storage Node settings I set the bitrate to 1000M. I’ve also compiled the latest kernel to make sure the latest patches are working (I’ve disabled SPECULATION_MITIGATIONS as I’ve read that this can affect CPU performance, and sometimes I find myself imaging computers with really weak CPUs). I also prepared my own init.xz with partclone 0.3.20, and then added these patches to the scripts and updated Buildroot to not have problems with BTRFS.

Also, this is how my current FOG configuration looks - as if anyone was curious what exactly I changed that made me achieve the speeds I did. A bit of an offtopic, but maybe it will be of use to someone.