Very slow cloning speed on specific model

Sebastian Roth

@Duncan I don’t have Acronis 2019 to test but I found an ISO download on their official website (dl.acronis.com) and booted that in a VM. On bootup I definitely see Linux kernel message and using Ctrl+Alt+F2 I as able to switch to a virtual terminal. Got kernel version 4.9.51-Acronis-b12-x86_64

We used kernel version 4.9.x in 2017 - see here: https://fogproject.org/kernels/

Though we don’t have the exact same kernel version yet we can still try to get close to that. We’ll still miss the Acronis specific patches (not sure but I guess they use some modifications) but it’s worth testing. We will build specific kernel and initrd for you.

Sebastian Roth

@Duncan Here is something to test:

sudo -i
cd /var/www/html/fog/service/ipxe
wget https://fogproject.org/kernels/bzImage-4.9.51
wget https://fogproject.org/inits/init-4.9.x.xz
chown apache:apache bzImage-4.9.51 init-4.9.x.xz

The last command will fail if your server is setup on Debian/Ubuntu rather than CentOS - use chown www-data:www-data ... instead.

Now go to the FOG web UI, edit the host settings of that particular slow notebook and set option Host Kernel to bzImage-4.9.51 and Host Init to init-4.9.x.xz. Schedule a new deploy task for it and see if there is any difference speed wise.

Duncan

@Sebastian-Roth

This is the same version as i have.

4.9.52-Acronis-b12-x86_64

Added the new Kernel and Init.

Getting a kernel panic error now.

XZ-compressed data is corrupt.

Sebastian Roth

@Duncan said in Very slow cloning speed on specific model:

Getting a kernel panic error now.
XZ-compressed data is corrupt.

Sorry, I thought you were on FOG 1.5.7 already. We changed the init size and if you are not on 1.5.7 yet you need to update the setting manually.

EDIT: Was going to add the details just now, but @Quazz was quicker
FOG web UI -> FOG configuration -> FOG settings -> TFTP Server -> KERNEL RAMDISK SIZE: change from 127000 to 256000

Quazz

@Duncan Update kernel ramdisk size under TFTP settings on the Web UI to 256000

Duncan

@Sebastian-Roth

still got same errors.

bumped it up to 512000 and it started to work. Then got

“No network interfaces found, your kernel is most probably missing the correct driver! Please check your network setup and try again!”

Please enter to continue.

Starts to load the script and then fails again on no network interface.

Sebastian Roth

@Duncan What kind of network card is in your machine? Please check out Windows device management for the PCI device IDs.

Duncan

@Sebastian-Roth

Intel I219-V

PCI\VEN_8086&DEV_15BE&SUBSYS_8549103C&REV_11
PCI\VEN_8086&DEV_15BE&SUBSYS_8549103C
PCI\VEN_8086&DEV_15BE&CC_020000
PCI\VEN_8086&DEV_15BE&CC_0200

Sebastian Roth

@Duncan Too bad, driver for Intel I219-V (8086:15be) was added with kernel version 4.12! Possibly I can build a 4.9.x version for you that has the driver in it. Though I am not sure this will actually lead to anything. This week is really busy and I am not sure when I will get to this. Will keep you posted.

Sebastian Roth

@Duncan After some deep digging in the kernel commit I found the ones that should add support for your NIC:

I will try to add those to 4.9.51 later on and let you know.

Sebastian Roth

@Duncan Just updated the kernel in https://fogproject.org/kernels/bzImage-4.9.51

Please re-download and try again.

Duncan

@Sebastian-Roth

Yasss!

Uploaded the kernel got some errors about kernel being to old.

On the client removed the “Host Init” setting but left the “Host Kernel” as bzImage-4.9.5.1 and kicked off an image.

Build at a nice decent speed now. Built 3 so far all at full speed!!

Thank you so much for all the help @Sebastian-Roth @Quazz

Sebastian Roth

@Duncan Sorry, but this doesn’t make any sense to me!

Using the new bzImage-4.9.51 you should definitely also need to use init-4.9.x.xz! Please take a picture of the settings in the web UI and post here. As well run ls -al /var/www/html/fog/service/ipxe/bzImage* /var/www/html/fog/service/ipxe/init* and post the full output here.

Build at a nice decent speed now. Built 3 so far all at full speed!!

Maybe it’s just some devices with drives that wouldn’t cause a problem using the latest kernel version as well???

Duncan

@Sebastian-Roth

Yeh, this is weird, i removed the settings from the other laptop i was building. And it still build… i guess it was a lucky laptop that could build.

Back to the original slow laptop, removed Host Kernel setting and its back to being slow.

Deleted original slow laptop from FOG, re registered it and added in bzImage-4.9.5.1 to host kernel. Host init still blank.

Deployed an image and its full speed.

 ls -al /var/www/html/fog/service/ipxe/bzImage* /var/www/html/fog/service/ipxe/init*
-rw-r--r-- 1 fog      fog       8118832 Dec 18  2018 /var/www/html/fog/service/i                                    pxe/bzImage
-rw-r--r-- 1 fog      fog       7562352 Dec 18  2018 /var/www/html/fog/service/i                                    pxe/bzImage32
-rw-r--r-- 1 fog      www-data  7465280 Jun 27  2017 /var/www/html/fog/service/i                                    pxe/bzImage32_OLD
-rw-r--r-- 1 www-data www-data  7942736 Dec  4 23:16 /var/www/html/fog/service/i                                    pxe/bzImage-4.9.51
-rw-r--r-- 1 fog      www-data  7601536 Jun 27  2017 /var/www/html/fog/service/i                                    pxe/bzImage_OLD
-rw-r--r-- 1 fog      www-data 18646084 Jun 27  2017 /var/www/html/fog/service/i                                    pxe/init_32.xz
-rw-r--r-- 1 www-data www-data 19744348 Dec  1 07:31 /var/www/html/fog/service/i                                    pxe/init-4.9.x.xz
-rw-r--r-- 1 fog      www-data 19605632 Jun 27  2017 /var/www/html/fog/service/i                                    pxe/init.xz

Duncan

@Duncan

Iv built a few more laptops now.

2 built straight out the box, no kernels or inits needed.

One had the slowness issue. In the host page i just added the kernel setting.

Deployed the image and away it went. Full speed. building about 8gb/min

Sebastian Roth

@Duncan Can’t believe it but if it’s the way you saying (and showing in the pictures) - what can I say…

@Quazz gave me a good hint on kernel 4.10 or 4.11 introducing APST. We kind of expect this to be causing the problem. See some information on this here: https://wiki.archlinux.org/index.php/Solid_state_drive/NVMe#Power_Saving_APST

He also just added the nvme cli tools to the FOS initrds so we could try to work on debugging more of this with more recent kernel versions.

It’s all up to you. If you are happy with the old 4.9.51 kernel we can just leave it like that. Though I don’t think it’s a great solution.

Quazz

@Duncan This suggests that it is indeed a kernel issue. Interesting that it ran at all with the newer inits since they’re only slated for backwards compatibility to 4.14 I believe.

My best guess at the moment is that the APST feature introduced in 4.10 is either the problem in its entirety or related to it somehow.

It’s still building, but when it’s done, there will be an init available at https://dev.fogproject.org/blue/organizations/jenkins/fos/detail/master/107/artifacts

EDIT: That build failed due to unrelated error, here is a different link https://drive.google.com/open?id=1u_HuN5NSpzb7YmQBAsrzDELteNmlWUWU

This will include an NVME cli utility that will give some info and allow some management over the NVME device.

I’d be interested in seeing a debug deploy on this init (use kernel 4.19 as well). If you could schedule one for a problematic host and run the following commands that would help a ton.

sudo nvme get-feature -f 0x0c -H /dev/nvme0

That will list out some info, the one I’m interested in is whether APST is enabled or not.

If it’s enabled you can disable it by doing

sudo nvme set-feature -f 0x0c -v=0 /dev/nvme0

Then type

fog

Press enter (you’ll have to do this a couple more times until it starts partclone and such)

I’m hoping that this will resolve the issue entirely and if so we can add to the inits if an NVME device is detected. APST is unneeded in FOS environment since we don’t care about power consumption of the storage device since it just needs to get captured or deployed and then the system takes it from there.

Duncan

@Quazz said in Very slow cloning speed on specific model:

Downloaded new Init_partclone added to host. Set the Kernel to 4.19.

Ran commands, APST was enabled. I disabled it and started to image.

Now its hung on Restoring Partition Tables GPT…

Rebooted, and tried again. Its now building at 2.7gb/min.

One thing i did notice was my storage nodes where on an old kernel 4.11.0. I have now copied over the latest ones to the nodes.

Quazz

@Duncan Thank you for trying it out. Very interesting results!

Much better than before, though not quite the speed you’d expect either.

@Sebastian-Roth What do you think? Should we investigate further?

Duncan

@Quazz

i can live with these speeds. image is only 70gb.

Alot faster than three weeks.

Im going to test this on my other sites now and see what speeds i get.

Seems to be that APST though. I wonder if some have it enabled out of the box and others dont. Im going to run the command to check on a “working” laptop and see if its disabled by default.

Very slow cloning speed on specific model

219

12.0k

17.3k

155.2k