Very slow cloning speed on specific model

[george1421]

@Quazz Would there be any advantage to building that command into the FOS scripts where / if a specific kernel parameter (i.e. a flag parameter) existed it would then signal the nvme-cli command to run during system startup? That way the kernel parameter could be set per machine or globally. That decision could then be up to the FOG Admin to use it or not. By default it would not run.

[Sebastian Roth]

@george1421 As we don’t fancy the NVMe energy saving modes I don’t see why we shouldn’t set this for everyone. Sure we don’t know the consequences yet but most of us seem pretty assured this is not causing harm but only helps. Keeping my fingers crossed. The more people test dev-branch the sooner we’ll know.

[Middle]

Unfortunately, nvme_core.default_ps_max_latency_us=0 hasn’t worked for me. I’ve tried both setting this manually via Host Kernel Argument using the default 1.5.7 kernel and updating to the latest dev-branch which seems to include it by default. Both result in slow transfer on an HP Elitebook 840 G6 (latest December BIOS).

Disabling APST using the init_partclone.xz and debug command that @Quazz posted gets an average transfer speed around 2.5GB/min which is a usable improvement. Is there a way to automate/improve this rather than entering debug mode each time to disable APST?

I haven’t been able to get the bzImage-4.9.51 and init-4.9.x.xz combo to work either (kernel panic with just bzImage-4.9.51 and partclone errors with both of them).

[Sebastian Roth]

@Middle said in Very slow cloning speed on specific model:

Is there a way to automate/improve this rather than entering debug mode each time to disable APST?

Sure, you can use post init scripts!

I am really confused on why you’d get the “kernel too old” error (even when it had already started to image - very strange)?! Unfortunately our build server is offline again and I can’t build another init-4.9.x.xz right now.

[Sebastian Roth]

@Middle Looking back at @Duncan’s first post in this topic I see those details:

EliteBook 840 G6 … Toshiba KXG60ZNV256G 79VA215DKRVN

Can you please check if you actually have the exact same disk model?

As well, can you please try using Acronis to capture and deploy an image just to see if their kernel works on your hardware. I will send you a link as private message. Check the speech bubble in the top right corner.

[Middle]

@Sebastian-Roth said in Very slow cloning speed on specific model:

Can you please check if you actually have the exact same disk model?

Just checked, the disk is different. It’s Micron M.2 NVMe Gen3 x4 Model: MTFDHBA256TCK and the HP part number: L36057-001.

I’ll give Acronis a try this morning.

By removing the disk and adding to a HP 840 G5, we can image without issues (I think Duncan had this as well).

[dylan123]

@Sebastian-Roth said in Very slow cloning speed on specific model:

@Duncan Just updated the kernel in https://fogproject.org/kernels/bzImage-4.9.51

Please re-download and try again.

I’ve been having a similar issue as the OP with one of these images. I’ve updated my fog to the latest version and got the latest kernel but I suppose that it doesn’t necessarily mean it gets the older ones.

I’ve ssh into my Fog environment, ran a command to download the kernel however I’m not sure what to do with it after that point?

[Sebastian Roth]

@dylan123 Now that you have downloaded the kernel make sure it’s in the right location (/var/www/html/fog/service/ipxe/), then edit the host’s settings in the web UI and set bzImage-4.9.51 as Host Kernel. Schedule a task, boot the device and pay attention to where it says bzImage-4.9.51... ok.

[dylan123]

@Sebastian-Roth said in Very slow cloning speed on specific model:

@dylan123 Now that you have downloaded the kernel make sure it’s in the right location (/var/www/html/fog/service/ipxe/), then edit the host’s settings in the web UI and set bzImage-4.9.51 as Host Kernel. Schedule a task, boot the device and pay attention to where it says bzImage-4.9.51... ok.

Thanks Seb.

Had some issues when I attempted to run the image.

Just showing I’ve got the files in the right area -

Have put the argument in -

When it boots, It says the Kernel is old and then it goes to this -

It then just counts up from /dev/nvme0n1p2, 3, etc

Without the init argument I get a different kind of issue which ends in -

Fatal : Kernel too old
Kernal panic - not syncing: Attempting to kill init! exitcode=0x00007f00

kernel Offset: disabled
—[ end Kernel panic - not syncing: Attempting to kill init! exitcode=0x00007f00

I’ve only got one more machine to go so I might just be better doing this one from scratch but figured I’d mention it anyways as I’m sure someone else will likely have a similar issue and if it’s something I can fix in the next couple of days I’ll do it that way before manually setting it up.

[Sebastian Roth]

@dylan123 We have seen this “late” message of “kernel too old” already and I still have no idea how it can happen that late in the process. When you leave the init parameter off it uses the current init file which shouldn’t be capable to play with the 4.9.51 kernel and ends in a Kernel panic. That’s understandable. But why the heck do we see the “kernel too old” when it already booted all the way into FOS and started cloning? @Quazz Any ideas?

Is you image set to “Partclone zstd” or “Partclone grip”??

[Quazz]

@Sebastian-Roth I’m guessing that some programs/functionality got coded with a correct glibc version for that kernel version, thus allowing it to boot and such, but that certain ones for whatever reason didn’t??? (partclone perhaps?) I have no clue how or why that would happen though.

That’s the only thing I can think of anyway!

edit: perhaps here https://github.com/Thomas-Tsai/partclone/blob/master/fail-mbr/compile-mbr.sh

It might accidentally call system GCC which might use a newer glibc version, thus causing the mismatch?

[dylan123]

@Sebastian-Roth said in Very slow cloning speed on specific model:

Is you image set to “Partclone zstd” or “Partclone grip”??

Can confirm I’m using partclone gzip

[Sebastian Roth]

@Quazz said in Very slow cloning speed on specific model:

edit: perhaps here https://github.com/Thomas-Tsai/partclone/blob/master/fail-mbr/compile-mbr.sh
It might accidentally call system GCC which might use a newer glibc version, thus causing the mismatch?

Hmm interesting idea!! Looking at this for a bit I can’t think of how this could make partclone hit the wall. The messge “Broken pipe” usually means that the program feeding the pipe crashed. In this case pigz. Though this should actually compile against the proper buildroot glibc (older version): https://github.com/FOGProject/fos/blob/master/Buildroot/package/pigz/pigz.mk

[Sebastian Roth]

@Quazz The error message is definitely thrown by glibc: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/dl-osinfo.h;hb=ef4e158c736d067304164c3daa763e4f425af248#l44

[Quazz]

@Sebastian-Roth Yes, glibc is generally the problem in kernel mismatches such as these, the question is where exactly? I’m not sure, but could it also say broken pipe if it gets interrupted unexpectedly by its piped data throwing a critical error such as say… kernel too old?

edit: Looking through build logs, ZSTD seems to be getting compiled against system gcc atm, at least from what I can tell. But OP is using pigz, so uhhhh

edit2: On the note of pigz, Buildroot provides a version these days, so we could remove the manually specified package unless we are attached to version 2.3.4 for some reason

edit3: Perhaps the workspace wasn’t cleaned properly for the build? Or alternatively, perhaps the GCC version has to be lowered.

[Sebastian Roth]

@Quazz Thanks!! I will try to build another init for 4.9.x kernels when our build server is back to life.

[darkxeno]

@Sebastian-Roth Im having the same issue We are tryign to image the new dell optiplex ultra 7070s and it has been hit or miss. It’s either really slow to even register the host and To even get the imaging process is slow I have one computer that has been running for 2 hours now and its just stuck on the first part clone screen.

[Sebastian Roth]

@darkxeno Build server is back and I started a fresh 4.9.x init build. Will update here soon.

[Sebastian Roth]

@darkxeno @dylan123 @Middle @oleg-knysh Build is done. Anyone keen to test?

sudo -i
cd /var/www/html/fog/service/ipxe
wget https://fogproject.org/kernels/bzImage-4.9.51
wget https://fogproject.org/inits/init-4.9.x.xz
chown apache:apache bzImage-4.9.51 init-4.9.x.xz

[Sebastian Roth]

@Quazz said:

edit2: On the note of pigz, Buildroot provides a version these days, so we could remove the manually specified package unless we are attached to version 2.3.4 for some reason

Don’t think we are bound to use 2.3.4 - not that I know of. I just pushed on to buildroot 2019.04.8 and their official pigz version 2.4.