Very slow cloning speed on specific model

Middle

@Sebastian-Roth said in Very slow cloning speed on specific model:

Can you please check if you actually have the exact same disk model?

Just checked, the disk is different. It’s Micron M.2 NVMe Gen3 x4 Model: MTFDHBA256TCK and the HP part number: L36057-001.

I’ll give Acronis a try this morning.

By removing the disk and adding to a HP 840 G5, we can image without issues (I think Duncan had this as well).

dylan123

@Sebastian-Roth said in Very slow cloning speed on specific model:

@Duncan Just updated the kernel in https://fogproject.org/kernels/bzImage-4.9.51

Please re-download and try again.

I’ve been having a similar issue as the OP with one of these images. I’ve updated my fog to the latest version and got the latest kernel but I suppose that it doesn’t necessarily mean it gets the older ones.

I’ve ssh into my Fog environment, ran a command to download the kernel however I’m not sure what to do with it after that point?

Sebastian Roth

@dylan123 Now that you have downloaded the kernel make sure it’s in the right location (/var/www/html/fog/service/ipxe/), then edit the host’s settings in the web UI and set bzImage-4.9.51 as Host Kernel. Schedule a task, boot the device and pay attention to where it says bzImage-4.9.51... ok.

dylan123

@Sebastian-Roth said in Very slow cloning speed on specific model:

@dylan123 Now that you have downloaded the kernel make sure it’s in the right location (/var/www/html/fog/service/ipxe/), then edit the host’s settings in the web UI and set bzImage-4.9.51 as Host Kernel. Schedule a task, boot the device and pay attention to where it says bzImage-4.9.51... ok.

Thanks Seb.

Had some issues when I attempted to run the image.

Just showing I’ve got the files in the right area -

Have put the argument in -

When it boots, It says the Kernel is old and then it goes to this -

It then just counts up from /dev/nvme0n1p2, 3, etc

Without the init argument I get a different kind of issue which ends in -

Fatal : Kernel too old
Kernal panic - not syncing: Attempting to kill init! exitcode=0x00007f00

kernel Offset: disabled
—[ end Kernel panic - not syncing: Attempting to kill init! exitcode=0x00007f00

I’ve only got one more machine to go so I might just be better doing this one from scratch but figured I’d mention it anyways as I’m sure someone else will likely have a similar issue and if it’s something I can fix in the next couple of days I’ll do it that way before manually setting it up.

Sebastian Roth

@dylan123 We have seen this “late” message of “kernel too old” already and I still have no idea how it can happen that late in the process. When you leave the init parameter off it uses the current init file which shouldn’t be capable to play with the 4.9.51 kernel and ends in a Kernel panic. That’s understandable. But why the heck do we see the “kernel too old” when it already booted all the way into FOS and started cloning? @Quazz Any ideas?

Is you image set to “Partclone zstd” or “Partclone grip”??

Quazz

@Sebastian-Roth I’m guessing that some programs/functionality got coded with a correct glibc version for that kernel version, thus allowing it to boot and such, but that certain ones for whatever reason didn’t??? (partclone perhaps?) I have no clue how or why that would happen though.

That’s the only thing I can think of anyway!

edit: perhaps here https://github.com/Thomas-Tsai/partclone/blob/master/fail-mbr/compile-mbr.sh

It might accidentally call system GCC which might use a newer glibc version, thus causing the mismatch?

dylan123

@Sebastian-Roth said in Very slow cloning speed on specific model:

Is you image set to “Partclone zstd” or “Partclone grip”??

Can confirm I’m using partclone gzip

Sebastian Roth

@Quazz said in Very slow cloning speed on specific model:

edit: perhaps here https://github.com/Thomas-Tsai/partclone/blob/master/fail-mbr/compile-mbr.sh
It might accidentally call system GCC which might use a newer glibc version, thus causing the mismatch?

Hmm interesting idea!! Looking at this for a bit I can’t think of how this could make partclone hit the wall. The messge “Broken pipe” usually means that the program feeding the pipe crashed. In this case pigz. Though this should actually compile against the proper buildroot glibc (older version): https://github.com/FOGProject/fos/blob/master/Buildroot/package/pigz/pigz.mk

Sebastian Roth

@Quazz The error message is definitely thrown by glibc: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/dl-osinfo.h;hb=ef4e158c736d067304164c3daa763e4f425af248#l44

Quazz

@Sebastian-Roth Yes, glibc is generally the problem in kernel mismatches such as these, the question is where exactly? I’m not sure, but could it also say broken pipe if it gets interrupted unexpectedly by its piped data throwing a critical error such as say… kernel too old?

edit: Looking through build logs, ZSTD seems to be getting compiled against system gcc atm, at least from what I can tell. But OP is using pigz, so uhhhh

edit2: On the note of pigz, Buildroot provides a version these days, so we could remove the manually specified package unless we are attached to version 2.3.4 for some reason

edit3: Perhaps the workspace wasn’t cleaned properly for the build? Or alternatively, perhaps the GCC version has to be lowered.

Sebastian Roth

@Quazz Thanks!! I will try to build another init for 4.9.x kernels when our build server is back to life.

darkxeno

@Sebastian-Roth Im having the same issue We are tryign to image the new dell optiplex ultra 7070s and it has been hit or miss. It’s either really slow to even register the host and To even get the imaging process is slow I have one computer that has been running for 2 hours now and its just stuck on the first part clone screen.

Sebastian Roth

@darkxeno Build server is back and I started a fresh 4.9.x init build. Will update here soon.

Sebastian Roth

@darkxeno @dylan123 @Middle @oleg-knysh Build is done. Anyone keen to test?

sudo -i
cd /var/www/html/fog/service/ipxe
wget https://fogproject.org/kernels/bzImage-4.9.51
wget https://fogproject.org/inits/init-4.9.x.xz
chown apache:apache bzImage-4.9.51 init-4.9.x.xz

Sebastian Roth

@Quazz said:

edit2: On the note of pigz, Buildroot provides a version these days, so we could remove the manually specified package unless we are attached to version 2.3.4 for some reason

Don’t think we are bound to use 2.3.4 - not that I know of. I just pushed on to buildroot 2019.04.8 and their official pigz version 2.4.

bberret

I’m noticing an issue as well. We just received some HP 840 G6 laptops and they came pre-loaded with bios version 01.03.00 and I was able to image them just fine. But then I decided to upgrade one of them to bios version 01.03.04 before imaging and now the imaging process is super slow. It also looks like bios version 01.03.00 isn’t available to download. So now I’m stuck waiting for a bios update I guess. I’ve talked to HP chat and they didn’t have 01.03.00 bios available to download.

Sebastian Roth

@bberret said in Very slow cloning speed on specific model:

I’ve talked to HP chat and they didn’t have 01.03.00 bios available to download.

Probably good to keep on asking HP (email, telephone, …) about this to make them aware of the problem.

george1421

@bberret Will you try something for us? In the fog configuration -> fog settings -> general settings page there is a KERNEL ARGS parameter . Will you place this value in that field nvme_core.default_ps_max_latency_us=0 and then try imaging again. You may see a warning about that variable during imaging, but ignore it. It is a spurious error that is fixed in the yet unreleased FOG v1.5.8. This settings tells the nvme drive to not go into low power mode during imaging.

bberret

@george1421 Thanks for the quick reply. I tried what you suggested and didn’t seem to help. And now my theory with it being an issue with only bios version 01.03.00 doesn’t seem to be holding true. Just unboxed another laptop and it seems to be having that exact same issue. This issue isn’t just slow imaging, it is slow loading pretty much everything after it loads ipxe.efi file. when its trying to load bzImage file (which usually takes 1 second) it is taking 15 minutes before that file is loaded.

george1421

@bberret So two changes (test) come to mind.

Does this computer have a bios mode? If so, as a test change it to bios mode to see if the bzImage and init.xz transfer speeds are normal.
Do you have an add on card (pci-e) that has pxe booting capabilities. The idea is to remove the onboard pxe firmware and network card. See if bzImage transfer is normal or not.
I thought about booting FOS linux from a usb stick, but you are telling me that imaging is not fast too. If imaging was fast but pxe booting was slow then I might point to iPXE as being a problem. But in this case booting for a usb stick will not mask/test the issue.

Off the top of my head I would either say network adapter or pxe / uefi firmware.

Very slow cloning speed on specific model

169

11.6k

17.1k

154.6k