Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized

[hlalex]

First, I apologize if this is a duplicate thread. I have searched for days and not found anything that resembles our current issue. We have been using FOG for several years to image approximately 100 computers in 5 different labs. Up to this point we have not encountered any major problems that we could not find a documented solution for.

Our most recent refresh presented us with Dell’s Flex Bay MiniSAS PCIe NVME SSDs. Fog does not recognize the drives when in the Flex Bay MiniSAS adapter, however when we swap the M.2 SSDs over to a standard PCIe adapter, Fog has no issues recognizing the drives. We are running FOG Version 1.5.4 and have tried the default Kernel and updated Kernels 4.17.0-addbnx 64 and 4.17.0 64 with no luck.

We have also attempted booting using Clonezilla Live 2.5.5-38, and received error: “nvme nvme0n: failed to set APST feature (-19)”. Per This Bug, we have tested several values for “nvme_core.default_ps_max_latency_us=” with no luck. The only unix bootable ISO that has successfully detected the drive is Fedora 27 live CD.

We are contemplating dissecting the Fedora Kernel and Grub patches to build a custom bzImage, but wanted to reach out to the community to see if anyone had alternate suggestions first.

More system information:
Dell Precision Tower 5820
512GB Hynix M.2 PCIe NVMe Class 40 SSD
Front PCIe FlexBay (Shows up as MiniSAS PCIe SSD-0 in BIOS)
Intel Integrated controller (R ST-e) with 1-2 Front FlexBay NVMe PCIe Drives

[Sebastian Roth]

Kernel option CONFIG_VMD added to the official repo. Will upload new 4.19.6 kernels soon.

[george1421]

@Sebastian-Roth I think its OK to leave out of the 32bit kernel since we are mainly seeing 32bit kernel used on older hardware and some “special” tablets that have 64bit processors but pinned in 32bit mode. This FlexBay/SAS hardware is super new. I can’t see it being back ported to older systems.
Somewhere in a change log we should note that the “Intel Volume Management Device Driver” kernel driver was specifically left out of the 32 bit kernel by design because it is only supported in 64 bit mode.

[Sebastian Roth]

@gehasia said in Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized:

after more than 60 kernel compilations, i found the faulty kernel config option :
Please activate CONFIG_VMD=y

Great stuff! Thanks for all the work you’ve put into this. I just worked on adding this to the config and noticed that it seems to be a 64 bit only thing. Is that possible? See here where CONFIG_X86_64 is explicitly is needed. And sure when I tried to add this option to the 32 bit kernel config I couldn’t. Does that make sense to you?

[gehasia]

@george1421

Thanks ;)

Here is the stock Kernel 4.19.4 with the TomElliott .config file and the CONFIG_VMD set to Yes. The FOG patches are not included (this is the stock kernel)
https://drive.google.com/file/d/1IjtnlavnjQEpgVTyPOSzdkoMQjyH4ZDF/view?usp=sharing

For those who wants to try quickly & report problems is there is any left.

I suppose it should resolve
https://forums.fogproject.org/topic/12658/uefi-dell-workstation-tower-flexbay-pcie-nvme-windows-10-fog-cannot-detect-pcie-nvme-ssd

[george1421]

@gehasia Well done!! It should be pretty easy for the devs to add that kernel module to the next kernel update.

[gehasia]

Hi,

after more than 60 kernel compilations, i found the faulty kernel config option :
Please activate CONFIG_VMD=y
Bus Options -> PCI Controller Driver -> Intel Volume Management Device Driver

[gehasia]

@Sebastian-Roth Thanks for the link, i was looking for the fog .config and patches in the repos !
I’ll start by compiling the stock fedora kernel from src-rpm (including fedora patches) to get a valid build system, then to compile a stock kernel with the fedora .config then the one from TomElliott with fog patches and see what happens at each step. I’ll try to get back here before end of the day !

[Sebastian Roth]

@gehasia You wanna follow the instructions on building your kernel https://wiki.fogproject.org/wiki/index.php/Build_TomElliott_Kernel

You will be done in 20 minutes…

[gehasia]

Hi everybody,

My company own one of these too. Same problem with fog which fail on nvme disk not found.

I just installed Fedora 29 which works correctly on it (i’m currently writing with it), i will try to compile a kernel with the fog config to add to my grub configuration and see what happens with the F29 userlands. Years i didn’t compiled a kernel, hope to do it quick ;).

If there is anything you want me to do, ask me, i will keep the computer with me until a solution is found, and can do whatever needed to help resolve this issue.

[hlalex]

@hlalex said in Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized:

@george1421 Ok, F27 still installs and boots fine. When upgrading to F28, it fails to reboot and drops to a shell (:/#).

@george1421 Since none of the fresh install methods were working with F28, I tried F27 to make sure there it still worked properly (to rule out any settings/hardware changes). After getting F27 installed and booted properly, I then attempted to upgrade to F28 using the GUI software manager. This resulted in the same failure of F28 to boot as the fresh install method.

The logs I posted are from a fresh install of F28 on a SATA installed in one of the hot swap bays. And I completely agree on the oddness of this adapter. I can only assume they are trying to getthe most from the advertised 2700MB/s read & 1390MB/s write speeds of these drives, but in an easier to manage platform.

[george1421]

@hlalex said in Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized:

Ok, F27 still installs and boots fine. When upgrading to F28, it fails to reboot and drops to a shell

So is these results form FC27 or FC28. I looked at the modules listed and they are found in your FC28. I still don’t understand why its working in FC27 and equally confusing why when you install on sata it doesn’t see the nvme drive, but the installer does. This IS a strange bit of hardware you have.

[hlalex]

F28_4.18.16-200 running from SATA drive, does not detect NVMe with lsblk or blkid…

4.18.16-200.fc28.x86_64.log
4.18.16-200.fc28.x86_64_lsmod.log

[hlalex]

@george1421 Ok, F27 still installs and boots fine. When upgrading to F28, it fails to reboot and drops to a shell (:/#).

blkid returns:

[  989.751061] Buffer I/O error on dev dm-0, logical block 13107184, async page read
[  989.751677] Buffer I/O error on dev dm-1, logical block 4092912, async page read

lsmod gives:

Module                     Size    Used By
hid_apple                 16384    0
nouveau                 2187264    1
video                     45056    1 nouveau
mxm_wmi                   16384    1 nouveau
i2c_algo_bit              16384    1 nouveau
drm_kms_helper           196608    1 nouveau
ttm                      126976    1 nouveau
drm                      475136    4 drm_kms_helper,ttm,nouveau 
e1000e                   282624    0 
nvme                      36864    1 
crc32c_intel              24576    0 
nvme_core                 81920    3 nvme
serio_raw                 16384    0 
wmi                       28672    2 mxm_wmi,nouveau 
uas                       69632    1 uas

journalctl | grep -i error :

kernel: print_req_error: I/O error, dev nvme0n1, sector 34844272
kernel: Buffer I/O error on dev dm-1, logical block 4092878, async page read 
# multiple iterations of ^ this error with different block #'s & dm-1/dm-0
systemd-fschk[564]: fsck: error 2 (No such file or directory) while executing fsck.ext2 for /dev/mapper/fedora-root
mount[569]: mount: /systoot: wrong fs type, bad option, bad superblock on /dev/mapper/fedora-root, missing codepage or helper program, or other error.

EDIT: Installing F28 to SATA drive now. I will pull logs and upload them when finished.

[hlalex]

@george1421 I am attempting to install directly to the NVMe flexbay. To rule out any extraneous variables I am going back to the F27 4.13.9-300 Live install that was previously working with this drive arrangement. If it installs and boots, then I will try upgrading to F28 and pull logs. Then I will go back to trying to get logs for F28.

[george1421]

@hlalex Oh so close. Are you installing to the flexbay nvme or a sata disk? Its promising that FC28 sees the disk. We just need to find out what module its loading. When you run some installers, when in installer mode you can press alt-f2 or crlt-f2 to open a new console window to see what is going on in the background. From your error it sounds like the proper driver isn’t loaded into the boot partition for FC28 to see the disk. (but that is only a guess)

[hlalex]

@george1421 Ok, I have tried both the live installer and net installer for F28. Both methods detect the drive during the installation process. After the reboot, I get dumped to a dracut shell complaining that /dev/mapper/xxx root and swap does not exist.

[hlalex]

@george1421 Sorry for the delay, I had a drive fail in an array that has been taking up a lot of my time trying.

I missed @matijn’s last post and it has been deleted now… I will try to get those logs uploaded today.

[Padcotton]

Haha, so many questions here.
And so little answers. Its actually rather funny how long this whole conversation has been going for.
Good luck in figuring this out !

[george1421]

@matijn OK I’ve read your post 6 times and I don’t understand what you are saying. Sorry I feel I need more coffee this morning.

Are you saying that the latest FOG kernel at 4.18.11 works with this flexbay NVMe?