@Sebastian-Roth Yes, glibc is generally the problem in kernel mismatches such as these, the question is where exactly? I’m not sure, but could it also say broken pipe if it gets interrupted unexpectedly by its piped data throwing a critical error such as say… kernel too old?

edit: Looking through build logs, ZSTD seems to be getting compiled against system gcc atm, at least from what I can tell. But OP is using pigz, so uhhhh

edit2: On the note of pigz, Buildroot provides a version these days, so we could remove the manually specified package unless we are attached to version 2.3.4 for some reason

edit3: Perhaps the workspace wasn’t cleaned properly for the build? Or alternatively, perhaps the GCC version has to be lowered.

To elaborate, I recall acpi=off being useful for rather old machines (who’d often have buggy implementations), but the exact opposite for newer machines who rely upon ACPI.

@Sebastian-Roth I’m guessing that some programs/functionality got coded with a correct glibc version for that kernel version, thus allowing it to boot and such, but that certain ones for whatever reason didn’t??? (partclone perhaps?) I have no clue how or why that would happen though.

That’s the only thing I can think of anyway!

edit: perhaps here https://github.com/Thomas-Tsai/partclone/blob/master/fail-mbr/compile-mbr.sh

It might accidentally call system GCC which might use a newer glibc version, thus causing the mismatch?

@Sebastian-Roth Do you reckon that https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/8.0_release_notes/rhel-8_0_0_release#file_systems_and_storage_4 will be a problem?

@Sebastian-Roth I see no harm in it, though I did run into cases where APST had to be explicitily disabled because the latency parameter wasn’t sufficient. But we can cross that bridge if it pops up.

@george1421 As far as I’m aware, all disabling APST does is lock the drive to its “highest power state”. Which for the purposes of FOS isn’t a bad choice if it would otherwise malfunction.

I don’t foresee a problem doing this for all NVME devices, but of course there might be instances we are unaware about currently where it does matter for something.

That said, FOS only runs for a little while, so odds of it being bad are very low.

@Duncan As far as I understand it’s only for specific drives on specific laptops (even amongst the same model), but it’s relatively widespread regardless.

Potentially slight firmware differences or the like.

Using that init file, you could add the command line that disables APST to images/dev/postinitscripts

@Duncan Thank you for trying it out. Very interesting results!

Much better than before, though not quite the speed you’d expect either.

@Sebastian-Roth What do you think? Should we investigate further?

@Duncan This suggests that it is indeed a kernel issue. Interesting that it ran at all with the newer inits since they’re only slated for backwards compatibility to 4.14 I believe.

My best guess at the moment is that the APST feature introduced in 4.10 is either the problem in its entirety or related to it somehow.

It’s still building, but when it’s done, there will be an init available at https://dev.fogproject.org/blue/organizations/jenkins/fos/detail/master/107/artifacts

EDIT: That build failed due to unrelated error, here is a different link https://drive.google.com/open?id=1u_HuN5NSpzb7YmQBAsrzDELteNmlWUWU

This will include an NVME cli utility that will give some info and allow some management over the NVME device.

I’d be interested in seeing a debug deploy on this init (use kernel 4.19 as well). If you could schedule one for a problematic host and run the following commands that would help a ton.

sudo nvme get-feature -f 0x0c -H /dev/nvme0

That will list out some info, the one I’m interested in is whether APST is enabled or not.

If it’s enabled you can disable it by doing

sudo nvme set-feature -f 0x0c -v=0 /dev/nvme0

Then type

fog

Press enter (you’ll have to do this a couple more times until it starts partclone and such)

I’m hoping that this will resolve the issue entirely and if so we can add to the inits if an NVME device is detected. APST is unneeded in FOS environment since we don’t care about power consumption of the storage device since it just needs to get captured or deployed and then the system takes it from there.

A quick look at https://www.php.net/manual/en/migration74.php makes me think there shouldn’t be too many issues, though without a code evaluation I can’t say for certain.