Very slow cloning speed on specific model

[DeRo93]

@Tom-Elliott

Hi,

i tried the Init.xz and the kernel argument. I updated FOG and BIOS from the Notebook. Even tried older BIOS Versions.

With “slow” its like the rate is going to 20 kb / min … it starts with 70 MB / min and then goes lower and lower… Its a Deploy…

SSD in this Notebook is a “sk hynix bc501 hfm256gdjtng-8310a” with the newest firmware.

[Tom Elliott]

@DeRo93 In what Mode is the drive being read to the system? From BIOS you may find something like SATA Operation.

Can you change this to AHCI and see if this helps improve the speeds a bit? Maybe change to ATA - IDE if it’s available?

[DeRo93]

@Tom-Elliott HP UEFI does not have AHCI - IDE Options in their UEFI Setup
HP says AHCI is activated by default on newer Models.

[Quazz]

@Tom-Elliott As far as I understand, this problem is occurring specifically on NVME SSDs

[Duncan]

Did anyone get this working?

I have a large batch of the EliteBook 840 G6.

Some of which build and other are extremely slow.

Looks like my batch has 2 versions or firmware versions.

The one that works is “Toshiba KXG60ZNV256G 79VA215DKRVN” and the other is the same drive but an older firmware by the looks of it.

Im in contact with HP to see about getting a firmware updater…

[Duncan]

Littlwe update.

So i swapped SSD out of a “slow” building laptop into a “working full speed” laptop.

So i thought the working laptop would now build slow, well it didnt. Building at full speed.

The other laptop with the SSD from the original full speed laptop is still building slow.

So maybe its not SSD related, but something on the motherboard?

[Quazz]

@Duncan Are the BIOS versions the same on these laptops? Settings too?

[Duncan]

@Quazz

Yes, iv updated to the latest 01.03.03 Rev.A, and my tech guys setup all the BIOS in the same way.

Im going to rip through the 2 laptops and reset BIOS, then make sure all settings are duplicated.

Going to see if i can pull some more hardware info and see if there are any differences there…

[Quazz]

@Duncan Perhaps they are using different NVME controllers for some reason. Should be interesting to check it out at least.

[Duncan]

@Quazz This is what im starting to think, be back with some findings soon

[Duncan]

Running HWInfo im failing to see any differences between the 2 laptops. Even in the BIOS all settings and hardware info look the same…

[Sebastian Roth]

@Duncan Can you schedule debug deploy task on both machines. When you get to the shell run lspci -nn and hdparm -i /dev/sda. Take pictures and post here.

[Duncan]

@Sebastian-Roth

[Quazz]

For some reason this reminded me of some of the earliest (and most explored) reports on slow deployments on certain system drive combinations.

https://github.com/Thomas-Tsai/partclone/issues/112

That user notes that if a partition is formatted and mounted on the target disk just prior to restore that it runs at expected speeds (but in FOS only if it’s not NTFS for whatever reason). Next attempt to restore will be slow again unless the same step is taken.

They then went on to try a different drive (different brand) and that one worked normally.

[Sebastian Roth]

@Duncan Thanks for the pictures. Though I think it was such a great idea because comparing the two listings as pictures is very much error prone. I reckon both listings are identical but can’t say for sure absolutely. And even then we still don’t know if one model might have just a slightly different revision of some component.

Is it only writing to disk or also reading from disk? Even if it doesn’t make sense for you to capture from the “slow” model, can you still give it a try to see!?

Also I wonder if you could live boot some Linux CD/DVD distro and do write/read tests as well.

dd if=/dev/zero of=/dev/nvme...
dd if=/dev/nvme... of=/dev/null

NOTICE: Be aware the first command will completely destroy the data on your drive!! Only do this on machines you don't have valuable data on and can re-image again.

Did you get to talk to HP about this issue? What do they say?

[george1421]

@Sebastian-Roth I realize we are discussion nvme drives here and only for a point of reference, but in 2017 I created a benchmark post to compare the differences in different technologies (one being disk subsystem) and its impact on imaging. https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast/6

From there I had these two commands:

… the simple disk baseline I’m using the following linux command to create a sequential 1GB file on disk and then to read it back. This process is designed to simulate the single unicast workload. The command used to write the 1GB file is this:
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=direct
The command to read it back is:
echo 3 | tee /proc/sys/vm/drop_caches && time dd if=/tmp/test1.img of=/dev/null bs=8k
The echo command is intended to disable the read cache so we get a true read back value.

[Duncan]

@Sebastian-Roth

root@ubuntu:/home/ubuntu# dd if=/dev/nvme0n1 of=/dev/null
500118192+0 records in
500118192+0 records out
256060514304 bytes (256 GB, 238 GiB) copied, 231.216 s, 1.1 GB/s
root@ubuntu:/home/ubuntu# dd if=/dev/zero of=/dev/nvme0n1
dd: writing to '/dev/nvme0n1': No space left on device
500118193+0 records in
500118192+0 records out
256060514304 bytes (256 GB, 238 GiB) copied, 2204.34 s, 116 MB/s

I’ve emailed my guy at HP and will see what they say.

Going to upload from “slow” laptop now and see what speeds i get.

[Quazz]

@Duncan Please replace the dots with the appropriate ending for the drive. (eg 0n1) and try again.

[george1421]

@Duncan Unless the "no space on device’ is causing this number disparity you have 1GB/s read and 116MB/s write speeds. The 116MB/s write speeds is just faster than a fast rotating disk speed. Those numbers are using a commercial linux distribution. (Please state the version number of ubuntu live you used)

I’m wondering now 2 things.

1. If you booted into FOS Linux in debug mode and ran the same commands would you see similar results?
2. I know there is no correlation with this, but if you booted into windows and then ran a tool like Crystal Disk Mark or ATTO to see if we have this wide differences between read and write speeds? Now this will be using the windows kernel and windows driver. This way we can see if its a windows/linux driver issue or if the hardware just performs this way.

[Quazz]

@george1421 Regardless, partclone is much slower for him than even this dd test. In the github issue thread I linked, they hypothesize that dd uses O_DIRECT and partclone does not, hence the difference.

It’s also worth noting that they had better speeds when the target was mounted, which live Ubuntu does automatically when it can.

Of course this is working under the assumption that it’s the exact same issue.