Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized

[hlalex]

@george1421 It connects directly to the MoBo beside the CPU slot, I posted a pic of the PCIE headers ~7 posts back.

[george1421]

@hlalex ok thank you for the clarity. Between your pictures and a few Dell youtube videos I understand how everything plugs in.

I’ve been trying to think of a solution to get us where we need to be. I’m wondering if you have time, can you install fedora 28 on that workstation? Then upgrade the kernel to 4.18.x if its not already there? https://fedoraproject.org/wiki/Kernel

If this works with fedora 28 and linux kernel 4.18.x then maybe we can reverse engineer what is missing. We’ll need you to run the same collection scripts you had before because that collected exactly what we needed. If it works the only additional output would be a lsmod command to find out what modules were currently loaded.

If you did not want to write to the nvme disks, installing fedora on a sata disk would also work as long as the nvme tray disk is installed and detected by the OS.

[matijn]

This post is deleted!

[george1421]

@matijn OK I’ve read your post 6 times and I don’t understand what you are saying. Sorry I feel I need more coffee this morning.

Are you saying that the latest FOG kernel at 4.18.11 works with this flexbay NVMe?

[Padcotton]

Haha, so many questions here.
And so little answers. Its actually rather funny how long this whole conversation has been going for.
Good luck in figuring this out !

[hlalex]

@george1421 Sorry for the delay, I had a drive fail in an array that has been taking up a lot of my time trying.

I missed @matijn’s last post and it has been deleted now… I will try to get those logs uploaded today.

[hlalex]

@george1421 Ok, I have tried both the live installer and net installer for F28. Both methods detect the drive during the installation process. After the reboot, I get dumped to a dracut shell complaining that /dev/mapper/xxx root and swap does not exist.

[george1421]

@hlalex Oh so close. Are you installing to the flexbay nvme or a sata disk? Its promising that FC28 sees the disk. We just need to find out what module its loading. When you run some installers, when in installer mode you can press alt-f2 or crlt-f2 to open a new console window to see what is going on in the background. From your error it sounds like the proper driver isn’t loaded into the boot partition for FC28 to see the disk. (but that is only a guess)

[hlalex]

@george1421 I am attempting to install directly to the NVMe flexbay. To rule out any extraneous variables I am going back to the F27 4.13.9-300 Live install that was previously working with this drive arrangement. If it installs and boots, then I will try upgrading to F28 and pull logs. Then I will go back to trying to get logs for F28.

[hlalex]

@george1421 Ok, F27 still installs and boots fine. When upgrading to F28, it fails to reboot and drops to a shell (:/#).

blkid returns:

[  989.751061] Buffer I/O error on dev dm-0, logical block 13107184, async page read
[  989.751677] Buffer I/O error on dev dm-1, logical block 4092912, async page read

lsmod gives:

Module                     Size    Used By
hid_apple                 16384    0
nouveau                 2187264    1
video                     45056    1 nouveau
mxm_wmi                   16384    1 nouveau
i2c_algo_bit              16384    1 nouveau
drm_kms_helper           196608    1 nouveau
ttm                      126976    1 nouveau
drm                      475136    4 drm_kms_helper,ttm,nouveau 
e1000e                   282624    0 
nvme                      36864    1 
crc32c_intel              24576    0 
nvme_core                 81920    3 nvme
serio_raw                 16384    0 
wmi                       28672    2 mxm_wmi,nouveau 
uas                       69632    1 uas

journalctl | grep -i error :

kernel: print_req_error: I/O error, dev nvme0n1, sector 34844272
kernel: Buffer I/O error on dev dm-1, logical block 4092878, async page read 
# multiple iterations of ^ this error with different block #'s & dm-1/dm-0
systemd-fschk[564]: fsck: error 2 (No such file or directory) while executing fsck.ext2 for /dev/mapper/fedora-root
mount[569]: mount: /systoot: wrong fs type, bad option, bad superblock on /dev/mapper/fedora-root, missing codepage or helper program, or other error.

EDIT: Installing F28 to SATA drive now. I will pull logs and upload them when finished.

[hlalex]

F28_4.18.16-200 running from SATA drive, does not detect NVMe with lsblk or blkid…

4.18.16-200.fc28.x86_64.log
4.18.16-200.fc28.x86_64_lsmod.log

[george1421]

@hlalex said in Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized:

Ok, F27 still installs and boots fine. When upgrading to F28, it fails to reboot and drops to a shell

So is these results form FC27 or FC28. I looked at the modules listed and they are found in your FC28. I still don’t understand why its working in FC27 and equally confusing why when you install on sata it doesn’t see the nvme drive, but the installer does. This IS a strange bit of hardware you have.

[hlalex]

@hlalex said in Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized:

@george1421 Ok, F27 still installs and boots fine. When upgrading to F28, it fails to reboot and drops to a shell (:/#).

@george1421 Since none of the fresh install methods were working with F28, I tried F27 to make sure there it still worked properly (to rule out any settings/hardware changes). After getting F27 installed and booted properly, I then attempted to upgrade to F28 using the GUI software manager. This resulted in the same failure of F28 to boot as the fresh install method.

The logs I posted are from a fresh install of F28 on a SATA installed in one of the hot swap bays. And I completely agree on the oddness of this adapter. I can only assume they are trying to getthe most from the advertised 2700MB/s read & 1390MB/s write speeds of these drives, but in an easier to manage platform.

[gehasia]

Hi everybody,

My company own one of these too. Same problem with fog which fail on nvme disk not found.

I just installed Fedora 29 which works correctly on it (i’m currently writing with it), i will try to compile a kernel with the fog config to add to my grub configuration and see what happens with the F29 userlands. Years i didn’t compiled a kernel, hope to do it quick ;).

If there is anything you want me to do, ask me, i will keep the computer with me until a solution is found, and can do whatever needed to help resolve this issue.

[Sebastian Roth]

@gehasia You wanna follow the instructions on building your kernel https://wiki.fogproject.org/wiki/index.php/Build_TomElliott_Kernel

You will be done in 20 minutes…

[gehasia]

@Sebastian-Roth Thanks for the link, i was looking for the fog .config and patches in the repos !
I’ll start by compiling the stock fedora kernel from src-rpm (including fedora patches) to get a valid build system, then to compile a stock kernel with the fedora .config then the one from TomElliott with fog patches and see what happens at each step. I’ll try to get back here before end of the day !

[gehasia]

Hi,

after more than 60 kernel compilations, i found the faulty kernel config option :
Please activate CONFIG_VMD=y
Bus Options -> PCI Controller Driver -> Intel Volume Management Device Driver

[george1421]

@gehasia Well done!! It should be pretty easy for the devs to add that kernel module to the next kernel update.

[gehasia]

@george1421

Thanks

Here is the stock Kernel 4.19.4 with the TomElliott .config file and the CONFIG_VMD set to Yes. The FOG patches are not included (this is the stock kernel)
https://drive.google.com/file/d/1IjtnlavnjQEpgVTyPOSzdkoMQjyH4ZDF/view?usp=sharing

For those who wants to try quickly & report problems is there is any left.

I suppose it should resolve
https://forums.fogproject.org/topic/12658/uefi-dell-workstation-tower-flexbay-pcie-nvme-windows-10-fog-cannot-detect-pcie-nvme-ssd

[Sebastian Roth]

@gehasia said in Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized:

after more than 60 kernel compilations, i found the faulty kernel config option :
Please activate CONFIG_VMD=y

Great stuff! Thanks for all the work you’ve put into this. I just worked on adding this to the config and noticed that it seems to be a 64 bit only thing. Is that possible? See here where CONFIG_X86_64 is explicitly is needed. And sure when I tried to add this option to the 32 bit kernel config I couldn’t. Does that make sense to you?