@Duncan This suggests that it is indeed a kernel issue. Interesting that it ran at all with the newer inits since they’re only slated for backwards compatibility to 4.14 I believe.
My best guess at the moment is that the APST feature introduced in 4.10 is either the problem in its entirety or related to it somehow.
It’s still building, but when it’s done, there will be an init available at https://dev.fogproject.org/blue/organizations/jenkins/fos/detail/master/107/artifacts
EDIT: That build failed due to unrelated error, here is a different link https://drive.google.com/open?id=1u_HuN5NSpzb7YmQBAsrzDELteNmlWUWU
This will include an NVME cli utility that will give some info and allow some management over the NVME device.
I’d be interested in seeing a debug deploy on this init (use kernel 4.19 as well). If you could schedule one for a problematic host and run the following commands that would help a ton.
sudo nvme get-feature -f 0x0c -H /dev/nvme0
That will list out some info, the one I’m interested in is whether APST is enabled or not.
If it’s enabled you can disable it by doing
sudo nvme set-feature -f 0x0c -v=0 /dev/nvme0
Press enter (you’ll have to do this a couple more times until it starts partclone and such)
I’m hoping that this will resolve the issue entirely and if so we can add to the inits if an NVME device is detected. APST is unneeded in FOS environment since we don’t care about power consumption of the storage device since it just needs to get captured or deployed and then the system takes it from there.