Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized

hlalex · Aug 7, 2018, 2:57 PM

First, I apologize if this is a duplicate thread. I have searched for days and not found anything that resembles our current issue. We have been using FOG for several years to image approximately 100 computers in 5 different labs. Up to this point we have not encountered any major problems that we could not find a documented solution for.

Our most recent refresh presented us with Dell’s Flex Bay MiniSAS PCIe NVME SSDs. Fog does not recognize the drives when in the Flex Bay MiniSAS adapter, however when we swap the M.2 SSDs over to a standard PCIe adapter, Fog has no issues recognizing the drives. We are running FOG Version 1.5.4 and have tried the default Kernel and updated Kernels 4.17.0-addbnx 64 and 4.17.0 64 with no luck.

We have also attempted booting using Clonezilla Live 2.5.5-38, and received error: “nvme nvme0n: failed to set APST feature (-19)”. Per This Bug, we have tested several values for “nvme_core.default_ps_max_latency_us=” with no luck. The only unix bootable ISO that has successfully detected the drive is Fedora 27 live CD.

We are contemplating dissecting the Fedora Kernel and Grub patches to build a custom bzImage, but wanted to reach out to the community to see if anyone had alternate suggestions first.

More system information:
Dell Precision Tower 5820
512GB Hynix M.2 PCIe NVMe Class 40 SSD
Front PCIe FlexBay (Shows up as MiniSAS PCIe SSD-0 in BIOS)
Intel Integrated controller (R ST-e) with 1-2 Front FlexBay NVMe PCIe Drives

george1421 · Aug 7, 2018, 3:54 PM

Do you have your storage setup as a raid?

If not we may need to do some data collection steps to find out a bit more about that hardware.

hlalex · Aug 7, 2018, 4:40 PM

Hey george,

We do not have raid on the FOG server storage, and as far as I can tell the 5820s do not either. I have tried [CTRL] + I during boot and nothing comes up. Also, in BIOS the only raid options I see pertain to SATA drives (also disabled), while the NVMe drive shows up under “PCIE Drives” > “MiniSAS PCIe SSD-0”.

Let me know what data you need and I will get it up.

george1421 · Aug 7, 2018, 6:05 PM

@hlalex ok next question, since you are using a dell, and can we assume that you are pxe booting into uefi mode? If so, go back into the firmware and change the disk mode from “Raid-on” to “AHCI” mode. Then pxe boot your computer. There is a documented issue with LINUX and the intel rapidstore hardware when raid-on is enabled. After imaging but before the first OS boot you can switch it back with no harm. Or just leave it in ahci mode and have no harm.

hlalex · Aug 7, 2018, 8:19 PM

We actually use Legacy mode. We have found it to be much easier to manage, especially with our dual-boot images (W10 + Fedora 27).

Switching to “AHCI” is one of our first changes before attempting to push an image. We also enable Legacy option ROMs and set the boot sequence to Legacy.

We are able to image the Hynix drives in the new 5820 systems after removing the M.2 from the 5820 FlexBay port and installing it in a 5810 PCIe adapter. This makes us think the hickup is related to the pxe/tftpboot/bzImage process detecting the drive due to some difference introduced by the MiniSAS/backplane.

Our typical BIOS changes:
Enable Legacy Option ROMs
Set boot option to Legacy
SATA Operation = AHCI
Disable Intel VMD Technology
TPM off
Secure Boot Disabled

george1421 · Aug 7, 2018, 8:31 PM

OK well, lets put everything back in the system and then schedule a debug deploy (tick the debug checkbox when you setup a deploy task). Then pxe boot the target computer. After a few key presses FOS will drop you to a command prompt on the target computer.

These next steps will make it easier for you to copy and paste in FOS.

Get the IP address of the FOS target computer with ip addr show. Record this value for future reference.
Give root a temporary password in FOS. Key in passwd and give root a simple password like hello. I might complain, but it will take the simple password.
Now use the free ssh program called putty to connect to the FOS engine using the IP address from step 1. Login as root with the password of hello from step 2.

Now from here you should have console access in the FOS engine. From here lets collect a bit of information.

First we will collect what devices are found in this system. Copy the results of this command into a reply in this thread:
lspci -nn
Lets collect the block devices FOS see with:
lsblk

We may need to review the /var/syslog but for now lets see what is returned with these commands.

hlalex · Aug 8, 2018, 11:49 AM

Ok with the system back in the original hardware configuration (BIOS settings as listed in my last comment):

lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:2020] (rev 04)
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
00:05.2 System peripheral [0880]: Intel Corporation Device [8086:2025] (rev 04)
00:05.4 PIC [0800]: Intel Corporation Device [8086:2026] (rev 04)
00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
00:14.0 USB controller [0c03]: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller [8086:a2af]
00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
00:17.0 RAID bus controller [0104]: Intel Corporation C600/X79 series chipset SATA RAID Controller [8086:2826]
00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
00:1c.6 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #7 [8086:a296] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2d3]
00:1f.2 Memory controller [0580]: Intel Corporation 200 Series PCH PMC [8086:a2a1]
00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
00:1f.4 SMBus [0c05]: Intel Corporation 200 Series PCH SMBus Controller [8086:a2a3]
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (5) I219-LM [8086:15e3]
02:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
16:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
16:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
64:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
64:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
64:08.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
64:09.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
64:0a.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
64:0a.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
64:0a.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
64:0a.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
64:0a.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
64:0a.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
64:0a.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
64:0a.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
64:0b.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
64:0b.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
64:0b.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
64:0b.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
64:0c.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
64:0c.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
64:0c.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
64:0c.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
64:0c.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
64:0c.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
64:0c.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
64:0c.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
64:0d.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
64:0d.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
64:0d.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
64:0d.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
65:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2000] [10de:1c30] (rev a1)
65:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
b2:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 04)
b2:03.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port D [8086:2033] (rev 04)
b2:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
b2:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)

lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb      8:16   1  3.8G  0 disk 
`-sdb1   8:17   1    1M  0 part

Due to the network configuration I have to use a custom iPXE rom, the 3.8G /dev/sdb that shows up is the USB drive with the iPXE bootloader

george1421 · Aug 8, 2018, 12:53 PM

@hlalex Well we know a few things now.

The only sata controller I see is:

00:17.0 RAID bus controller [0104]: Intel Corporation C600/X79 series chipset SATA RAID Controller [8086:2826]

But lsblk doesn’t see any drives in that system. On the host configuration for this system, add the following kernel parameter and reboot back into the debug deploy.
mdraid=true

I have a tutorial on how to configure FOS to see an intel raid configuraiton: https://forums.fogproject.org/topic/7882/capture-deploy-to-target-computers-using-intel-rapid-storage-onboard-raid

I seriously don’t think this will solve your issue because we can’t see any existing disks at the moment.

Since you have fedora working on this system, maybe we can use that to reverse engineer how Fedora make it work.

Boot into fedora and at the linux command prompt key in
sudo lspci -k -nn and post the output here. This will show the pci device with the kernel driver that is controlling it.

hlalex · Aug 8, 2018, 7:19 AM

EDIT: This was from a Fedora 27 Live CD.

I think the very last entry is what we are looking for.

sudo lspci -k -nn
00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:05.2 System peripheral [0880]: Intel Corporation Device [8086:2025] (rev 04)
00:05.4 PIC [0800]: Intel Corporation Device [8086:2026] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:14.0 USB controller [0c03]: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller [8086:a2af]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: xhci_hcd
00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
	Subsystem: Dell Device [1028:0738]
00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: mei_me
	Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1c.6 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #7 [8086:a296] (rev f0)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2d3]
	Subsystem: Dell Device [1028:0738]
00:1f.2 Memory controller [0580]: Intel Corporation 200 Series PCH PMC [8086:a2a1]
	Subsystem: Dell Device [1028:0738]
00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
00:1f.4 SMBus [0c05]: Intel Corporation 200 Series PCH SMBus Controller [8086:a2a3]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (5) I219-LM [8086:15e3]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: e1000e
	Kernel modules: e1000e
02:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
	Kernel modules: shpchp
16:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
16:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
64:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
64:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:08.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:09.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0a.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0a.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0a.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0b.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0b.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0b.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0b.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0c.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0c.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0d.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0d.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0d.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0d.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
	Subsystem: Dell Device [1028:0738]
65:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2000] [10de:1c30] (rev a1)
	Subsystem: Dell Device [1028:11b3]
	Kernel driver in use: nouveau
	Kernel modules: nouveau
65:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
	Subsystem: Dell Device [1028:11b3]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
b2:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 04)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
b2:03.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port D [8086:2033] (rev 04)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
b2:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
b2:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]
	Subsystem: Device [1c5c:1527]
	Kernel driver in use: nvme
	Kernel modules: nvme

Just for completeness:

lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0   1.4G  1 loop 
loop1         7:1    0   6.5G  1 loop 
├─live-rw   253:0    0   6.5G  0 dm   /
└─live-base 253:1    0   6.5G  1 dm   
loop2         7:2    0    32G  0 loop 
└─live-rw   253:0    0   6.5G  0 dm   /
sdb           8:16   1  14.6G  0 disk 
└─sdb1        8:17   1  14.6G  0 part /run/initramfs/live
sr0          11:0    1  1024M  0 rom  
nvme0n1     259:0    0   477G  0 disk 
├─nvme0n1p1 259:1    0   650M  0 part 
├─nvme0n1p2 259:2    0   128M  0 part 
├─nvme0n1p3 259:3    0   499M  0 part 
└─nvme0n1p4 259:4    0 475.7G  0 part

george1421 · Aug 8, 2018, 1:26 PM

@hlalex I think you are spot on with your kernel driver name. The lsblk command from fedora is what I should expect from FOS.

One last thing I should have asked for with fedora, will you provide the output of sudo uname -a That will tell us what kernel they are using.

hlalex · Aug 8, 2018, 1:35 PM

Linux localhost-live 4.13.9-300.fc27.x86_64 #1 SMP Mon Oct 23 13:41:58 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

george1421 · Aug 8, 2018, 3:25 PM

@hlalex Well after some research, I think I found what is missing. We may need to get the developers involved to recompile the FOS kernel.

First some background.
FC27
Kernel used 4.13.9
Device discovered

b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]
	Subsystem: Device [1c5c:1527]
	Kernel driver in use: nvme
	Kernel modules: nvme

FOS 1.5.4
Kernel used 4.17.0 (or 4.15.2)
Device b3:00.0 not found/kernel driver not available.

Linux kernel config file found here: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config

Setting disabled is on line 2925: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config#L2925

#  CONFIG_NVMEM is not set

CONFIG_NVMEM == "Support for NVMEM(Non Volatile Memory) devices like EEPROM, EFUSES…

This framework is designed to provide a generic interface to NVMEM from both the Linux Kernel and the userspace.

This driver can also be built as a module. If so, the module will be called nvmem_core."

@Developers Opinion here? Is there a possibility to get this option turned on for linux kernels 4.15.2 and 4.17.0 for testing purposes?

hlalex · Aug 8, 2018, 6:46 PM

I just received word from Dell that they are going to swap the FlexBay Carriers out for standard PCIe adapters (similar model to the adapters in the Precision 5810s) that should hopefully work without issues.
I would like to provide as much information as I can before I have to ship them out, so please let me know what other logs/data/tests we may need to work out the issue. I’m sure these drives will become more prevalent in the near future and would like to help as much as I can. Not to mention I have another 40 system lab to refresh next summer, and I would love to have a solution if they wind up with this same drive configuration.

george1421 · Aug 8, 2018, 6:57 PM

@hlalex We surely like to get this addressed before you swap out the hardware. I’m sure others will either order the same hardware as you have, or other manufacturers will use this hardware design. It needs to be fixed.

hlalex · Aug 8, 2018, 7:59 PM

@george1421 Agreed. It looks like the replacements will not ship until 8/14 so we have a bit of time for testing.

What baffles me is that I have been in contact with multiple Dell Reps trying to determine exactly which hardware needs to be returned, and none have been able to answer what part is what or point me to any product information for this drive type. As far as I can tell from their site, this particular drive configuration is not currently offered as a retail addition. The specs related to the drive based on the dell support Service Tag lookup lists only:

414-BBBV : PCIe SSD (Front PCIe FlexBay)
Part Number Quantity Description
5G90D 1 INFORMATION, FRONT, PERIPHERAL COMPONENT INTERCONNECT EXPRESS , SOFTWARE SUPPORT DISKETTE, BOOT

401-ABJT : M.2 512GB PCIe NVMe Class 40 S olid State Drive
Part Number Quantity Description
XMW6J 1 SSDR, 512G, P34, 80S3, HYNIX, PC401

400-AVDR : Dell M.2 carrier
Part Number Quantity Description
66XHV 1 ASSEMBLY, DRIVE, BAY (DRIVE BAY), M.2, MODULE

I have looked high and low for any information on these part numbers with no luck.

george1421 · Aug 9, 2018, 12:25 AM

@hlalex I went ahead and compiled the latest FOS kernel with the suspected missing module enabled. This is kernel version 4.17.13. Understand its not the official FOS kernel because it doesn’t have the customized patches that FOG kernel developers add, but should work for our test to see if we can init that nvme memory device.

I’ll IM you a link to the test kernel.

For this test download the linked kernel to /var/www/html/fog/service/ipxe on the fog server, leave the name as bzImage41713 Then go into the host record for one of these test systems and set the “Host Kernel” parameter to bzImage41713. Then finally pxe boot the target system into debug capture task. Run the lspci -nn command and see if we can detect

b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]

If its in the list then run the lsblk command to see if has something we can mount.

hlalex · Aug 9, 2018, 3:10 PM

Good Morning @george1421. I stopped by the office to test the custom kernel (I’m going to pick up breakfast for my wife on my way home so she should be happy ;). No luck. The first attempt to boot gave init error, and I realized it was trying init_32.xz instead of init.xz. After adding init.xz to the "Host Init " parameter in the host record it booted into debug without issue. Bad news is that

b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]

does not show up with lspci -nn and only /dev/sdb (pxe boot usb) shows up with lsblk.

george1421 · Aug 9, 2018, 3:10 PM

@hlalex Well, digging a bit deeper into this…

Going line by line…

FC27 I see that the disk controller is in SATA mode

00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ahci

FOS the disk controllers was in Raid-on mode

00:17.0 RAID bus controller [0104]: Intel Corporation C600/X79 series chipset SATA RAID Controller [8086:2826]

Note that the hardware ID changes between the two modes. (not pointing fingers here) But as I said before linux does not see the disks behind raid-on devices in uefi mode. Even if you are in bios mode, please change the value to ahci mode and pull the lspci from FOS. I don’t believe that lspci under FOS supports the -k prompt, but that info would be handy.

I also found another kernel module that was disabled called “Devices/Memory Controller drivers”. After a bit more research I’m going to compile and updated kernel with Memory Controller support enabled.

hlalex · Aug 9, 2018, 4:17 PM

@george1421 I had noticed that as well, and double checked the Raid settings. It had been turned back on at some point (probably while I was going through everything else). I turned it back off and re-did the debug task with the same results.

It appears that the NVMe PCIe drives use an entirely separate controller from SATA drives. Apparently the specific controller/driver varies based on the interface between the MoBo and SSD. FOS can see the M.2 drives when the are connected via a standard PCIe adapter card, however when the same M.2 drive is connected via the FlexBay MiniSAS backplane, FOS no longer recognizes the drive. This holds true regardless of the SATA controller configuration–the drives are visible with both Raid & AHCI modes.

I found a dell support article that gives some good information about the differences between NVMe and AHCI here.

I checked through the FOS kernel, and found NVME Support is enabled in lines 880-890: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config#L880

#
# NVME Support
#
CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_FABRICS=y
CONFIG_NVME_FC=y
CONFIG_NVME_TARGET=y
CONFIG_NVME_TARGET_LOOP=y
CONFIG_NVME_TARGET_FC=y
CONFIG_NVME_TARGET_FCLOOP=y

Which explains why the drives are detected when plugged into a PCIe adapter. The MiniSAS connection is a completely different beast–or at least uses a different controller from the standard NVMe drive.

I will run through the debug again (with Raid off in bios) and post the results of lspci -nn, lsblk, uname -a, and anything else you suggest. I will also do the same thing with the M.2 plugged in through a PCIe adapter so we have comparison data.

george1421 · Aug 9, 2018, 4:21 PM

@hlalex I have 2 additional kernels for you to test, but I will only share with you tomorrow since you should have other activities planned for today.

Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized

170

12.0k

17.3k

155.2k