@DeRo93 We didn’t get the issue fixed but ran out of G6 laptops to test with. We’re expecting the next batch of 50 on Tuesday so I expect to be more active on here next week.

@Sebastian-Roth said in Very slow cloning speed on specific model:

Can you please check if you actually have the exact same disk model?

Just checked, the disk is different. It’s Micron M.2 NVMe Gen3 x4 Model: MTFDHBA256TCK and the HP part number: L36057-001.

I’ll give Acronis a try this morning.

By removing the disk and adding to a HP 840 G5, we can image without issues (I think Duncan had this as well).

@nrg The following is based on using Fog 1.5.7 master branch on CentOS.

Are you using the updated init_partclone.xz? Here’s the link:

https://drive.google.com/open?id=1u_HuN5NSpzb7YmQBAsrzDELteNmlWUWU

You need to copy this to /var/www/html/fog/service/ipxe and then run chown apache:apache init_partclone.xz to update file permissions.

We only have G6 laptops for imaging now, so I made this the default init under Fog Config > Fog Settings > TFTP Server > PXE BOOT IMAGE > enter init_partclone.xz. Alternatively, if you register hosts, you can add it to the host init section for just the G6 laptops.

You then have to edit the file /images/dev/postinitscripts/fog.postinit and add the line nvme set-feature -f 0x0c -v=0 /dev/nvme0

@nrg The following is based on using Fog 1.5.7 master branch on CentOS.

Are you using the updated init_partclone.xz? Here’s the link:

https://drive.google.com/open?id=1u_HuN5NSpzb7YmQBAsrzDELteNmlWUWU

You need to copy this to /var/www/html/fog/service/ipxe and then run chown apache:apache init_partclone.xz to update file permissions.

We only have G6 laptops for imaging now, so I made this the default init under Fog Config > Fog Settings > TFTP Server > PXE BOOT IMAGE > enter init_partclone.xz. Alternatively, if you register hosts, you can add it to the host init section for just the G6 laptops.

You then have to edit the file /images/dev/postinitscripts/fog.postinit and add the line nvme set-feature -f 0x0c -v=0 /dev/nvme0

@Quazz That’s great - setting the nvme command in the fog.postinit file and making the init-partclone.xz you created the system wide default means we can now just pxe boot and select deploy image without issues. Huge improvement. Many thanks.

Edit: for clarification, it’s the nvme set-feature -f 0x0c -v=0 /dev/nvme0 command I’m using, not the latency one.

@Quazz We’ve tried that as well but doesn’t help. I think it’s also included in the Dev branch by default now which we’re tried.

@Sebastian-Roth said in Very slow cloning speed on specific model:

@darkxeno @dylan123 @Middle @oleg-knysh Build is done. Anyone keen to test?

sudo -i
cd /var/www/html/fog/service/ipxe
wget https://fogproject.org/kernels/bzImage-4.9.51
wget https://fogproject.org/inits/init-4.9.x.xz
chown apache:apache bzImage-4.9.51 init-4.9.x.xz

Sorry for the late update. This didn’t change anything I’m afraid. I’ve been a little reluctant in updating during testing as the results I’ve had have been very inconsistent. I can’t explain it, but the first deployment of the day works without issues - it’s happened too many times now to be a coincidence.

I’ve currently changed back to the master branch to clean-up the server of the test kernels/inits we’ve been using. The only consistent deploy I can get is using the init_partclone.xy that @Quazz posted on the 5th Dec, entering debug mode and running the following:

nvme set-feature -f 0x0c -v=0 /dev/nvme0

This works every time. The average transfer speed is around 3GB/min, rather than >10GB/min that I get from the master branch build when it randomly works, but that’s good enough for us as the result is consistent and the image is small anyway.

This is google drive link Quazz provided for the init: https://drive.google.com/open?id=1u_HuN5NSpzb7YmQBAsrzDELteNmlWUWU

I’ve had a look at the post init script option to see if I can automate this rather than entering debug, but I’m not really sure what I’m doing here.

@Sebastian-Roth said in Very slow cloning speed on specific model:

Can you please check if you actually have the exact same disk model?

Just checked, the disk is different. It’s Micron M.2 NVMe Gen3 x4 Model: MTFDHBA256TCK and the HP part number: L36057-001.

I’ll give Acronis a try this morning.

By removing the disk and adding to a HP 840 G5, we can image without issues (I think Duncan had this as well).

Unfortunately, nvme_core.default_ps_max_latency_us=0 hasn’t worked for me. I’ve tried both setting this manually via Host Kernel Argument using the default 1.5.7 kernel and updating to the latest dev-branch which seems to include it by default. Both result in slow transfer on an HP Elitebook 840 G6 (latest December BIOS).

Disabling APST using the init_partclone.xz and debug command that @Quazz posted gets an average transfer speed around 2.5GB/min which is a usable improvement. Is there a way to automate/improve this rather than entering debug mode each time to disable APST?

I haven’t been able to get the bzImage-4.9.51 and init-4.9.x.xz combo to work either (kernel panic with just bzImage-4.9.51 and partclone errors with both of them).

@DeRo93 We didn’t get the issue fixed but ran out of G6 laptops to test with. We’re expecting the next batch of 50 on Tuesday so I expect to be more active on here next week.

@Quazz No change I’m afraid with the pcie_aspm args (slow transfer rate). I didn’t spot any errors like we get with the latency one, however I do receive ‘is an unknown key’ when trying to add in debug mode. I’ve tired with both the 5.1.16 and 4.19.64 kernels.

Incidentally, if I have a kernel args set and I use debug mode, it always seems to stop at the ‘Restoring Partition Tables (GPT)’ section. Running a normal deploy at least moves onto the Partclone screen and eventually to a slow transfer rate.

I’ve also installed the Sept 27th HP BIOS and Firmware pack. I’m still looking for a firmware update specifically for the disk.

@george1421 Back in the office and just tried this. We get unknown key for the latency parameter.

grep nvme doesn’t return the value either and running fog gets stuck at the ‘Restoring Partition Tables (GPT)’ section, so didn’t get as far as Partclone. Note we’re using Kernel 5.1.16.

Thanks for helping out.

@george1421 sysctl exists and returns results. Using sysctl -a | more we still don’t see anything that’s setting the latency parameter.