Very slow cloning speed on specific model

Sebastian Roth

@Quazz said in Very slow cloning speed on specific model:

As I understand it, Acronis uses WinPE, supporting the idea that this some kind of Linux problem, but it appears to be hard to track down exactly how and why it happens.

When you first mentioned Acronis I didn’t think much further. But now that you say WinPE I had to look this up. Not sure if newer versions are WinPE based but the older ones all were Linux based: https://kb.acronis.com/content/1537

@Duncan Which version of Acronis do you have? Any chance you can get to a console and run uname -a to get the kernel version for us?

Duncan

@Sebastian-Roth

Happy to help out where i can, we also have another batch of 60 coming. Will be interesting to see the success rate of these ones.

When i spoke to HP he was unsure of how to move forward with this. Hopefully gets escalated to there tech guys. I asked about and firmware updates, but he said only the BIOS was available.

Duncan

Acronis True Image 2019

cant see a way to drop into console to run that command.

Sebastian Roth

@Duncan said:

Acronis True Image 2019

Ok, reading some more on the web I found an article saying that Acronis actually has two different bootable medias - one WinPE/WinRE-based and one Linux-based. Do you have Acronis installed on one of your Windows PCs? Please see if you can launch the “Rescue Media Builder” as described here and create a Linux-based boot media. Just want to make sure we don’t compare apples with pears here when we see Acronis doing it fast.

Switching between virtual terminals in Linux is usually done by pressing Ctrl+Alt+F1 or F2…F7 - see if that gets you to a console.

Sebastian Roth

@Duncan I don’t have Acronis 2019 to test but I found an ISO download on their official website (dl.acronis.com) and booted that in a VM. On bootup I definitely see Linux kernel message and using Ctrl+Alt+F2 I as able to switch to a virtual terminal. Got kernel version 4.9.51-Acronis-b12-x86_64

We used kernel version 4.9.x in 2017 - see here: https://fogproject.org/kernels/

Though we don’t have the exact same kernel version yet we can still try to get close to that. We’ll still miss the Acronis specific patches (not sure but I guess they use some modifications) but it’s worth testing. We will build specific kernel and initrd for you.

Sebastian Roth

@Duncan Here is something to test:

sudo -i
cd /var/www/html/fog/service/ipxe
wget https://fogproject.org/kernels/bzImage-4.9.51
wget https://fogproject.org/inits/init-4.9.x.xz
chown apache:apache bzImage-4.9.51 init-4.9.x.xz

The last command will fail if your server is setup on Debian/Ubuntu rather than CentOS - use chown www-data:www-data ... instead.

Now go to the FOG web UI, edit the host settings of that particular slow notebook and set option Host Kernel to bzImage-4.9.51 and Host Init to init-4.9.x.xz. Schedule a new deploy task for it and see if there is any difference speed wise.

Duncan

@Sebastian-Roth

This is the same version as i have.

4.9.52-Acronis-b12-x86_64

Added the new Kernel and Init.

Getting a kernel panic error now.

XZ-compressed data is corrupt.

Sebastian Roth

@Duncan said in Very slow cloning speed on specific model:

Getting a kernel panic error now.
XZ-compressed data is corrupt.

Sorry, I thought you were on FOG 1.5.7 already. We changed the init size and if you are not on 1.5.7 yet you need to update the setting manually.

EDIT: Was going to add the details just now, but @Quazz was quicker
FOG web UI -> FOG configuration -> FOG settings -> TFTP Server -> KERNEL RAMDISK SIZE: change from 127000 to 256000

Quazz

@Duncan Update kernel ramdisk size under TFTP settings on the Web UI to 256000

Duncan

@Sebastian-Roth

still got same errors.

bumped it up to 512000 and it started to work. Then got

“No network interfaces found, your kernel is most probably missing the correct driver! Please check your network setup and try again!”

Please enter to continue.

Starts to load the script and then fails again on no network interface.

Sebastian Roth

@Duncan What kind of network card is in your machine? Please check out Windows device management for the PCI device IDs.

Duncan

@Sebastian-Roth

Intel I219-V

PCI\VEN_8086&DEV_15BE&SUBSYS_8549103C&REV_11
PCI\VEN_8086&DEV_15BE&SUBSYS_8549103C
PCI\VEN_8086&DEV_15BE&CC_020000
PCI\VEN_8086&DEV_15BE&CC_0200

Sebastian Roth

@Duncan Too bad, driver for Intel I219-V (8086:15be) was added with kernel version 4.12! Possibly I can build a 4.9.x version for you that has the driver in it. Though I am not sure this will actually lead to anything. This week is really busy and I am not sure when I will get to this. Will keep you posted.

Sebastian Roth

@Duncan After some deep digging in the kernel commit I found the ones that should add support for your NIC:

I will try to add those to 4.9.51 later on and let you know.

Sebastian Roth

@Duncan Just updated the kernel in https://fogproject.org/kernels/bzImage-4.9.51

Please re-download and try again.

Duncan

@Sebastian-Roth

Yasss!

Uploaded the kernel got some errors about kernel being to old.

On the client removed the “Host Init” setting but left the “Host Kernel” as bzImage-4.9.5.1 and kicked off an image.

Build at a nice decent speed now. Built 3 so far all at full speed!!

Thank you so much for all the help @Sebastian-Roth @Quazz

Sebastian Roth

@Duncan Sorry, but this doesn’t make any sense to me!

Using the new bzImage-4.9.51 you should definitely also need to use init-4.9.x.xz! Please take a picture of the settings in the web UI and post here. As well run ls -al /var/www/html/fog/service/ipxe/bzImage* /var/www/html/fog/service/ipxe/init* and post the full output here.

Build at a nice decent speed now. Built 3 so far all at full speed!!

Maybe it’s just some devices with drives that wouldn’t cause a problem using the latest kernel version as well???

Duncan

@Sebastian-Roth

Yeh, this is weird, i removed the settings from the other laptop i was building. And it still build… i guess it was a lucky laptop that could build.

Back to the original slow laptop, removed Host Kernel setting and its back to being slow.

Deleted original slow laptop from FOG, re registered it and added in bzImage-4.9.5.1 to host kernel. Host init still blank.

Deployed an image and its full speed.

 ls -al /var/www/html/fog/service/ipxe/bzImage* /var/www/html/fog/service/ipxe/init*
-rw-r--r-- 1 fog      fog       8118832 Dec 18  2018 /var/www/html/fog/service/i                                    pxe/bzImage
-rw-r--r-- 1 fog      fog       7562352 Dec 18  2018 /var/www/html/fog/service/i                                    pxe/bzImage32
-rw-r--r-- 1 fog      www-data  7465280 Jun 27  2017 /var/www/html/fog/service/i                                    pxe/bzImage32_OLD
-rw-r--r-- 1 www-data www-data  7942736 Dec  4 23:16 /var/www/html/fog/service/i                                    pxe/bzImage-4.9.51
-rw-r--r-- 1 fog      www-data  7601536 Jun 27  2017 /var/www/html/fog/service/i                                    pxe/bzImage_OLD
-rw-r--r-- 1 fog      www-data 18646084 Jun 27  2017 /var/www/html/fog/service/i                                    pxe/init_32.xz
-rw-r--r-- 1 www-data www-data 19744348 Dec  1 07:31 /var/www/html/fog/service/i                                    pxe/init-4.9.x.xz
-rw-r--r-- 1 fog      www-data 19605632 Jun 27  2017 /var/www/html/fog/service/i                                    pxe/init.xz

Duncan

@Duncan

Iv built a few more laptops now.

2 built straight out the box, no kernels or inits needed.

One had the slowness issue. In the host page i just added the kernel setting.

Deployed the image and away it went. Full speed. building about 8gb/min

Sebastian Roth

@Duncan Can’t believe it but if it’s the way you saying (and showing in the pictures) - what can I say…

@Quazz gave me a good hint on kernel 4.10 or 4.11 introducing APST. We kind of expect this to be causing the problem. See some information on this here: https://wiki.archlinux.org/index.php/Solid_state_drive/NVMe#Power_Saving_APST

He also just added the nvme cli tools to the FOS initrds so we could try to work on debugging more of this with more recent kernel versions.

It’s all up to you. If you are happy with the old 4.9.51 kernel we can just leave it like that. Though I don’t think it’s a great solution.

Very slow cloning speed on specific model

297

11.7k

17.2k

154.7k