• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized

    Scheduled Pinned Locked Moved
    Hardware Compatibility
    9
    124
    40.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H
      hlalex
      last edited by

      Hey george,

      We do not have raid on the FOG server storage, and as far as I can tell the 5820s do not either. I have tried [CTRL] + I during boot and nothing comes up. Also, in BIOS the only raid options I see pertain to SATA drives (also disabled), while the NVMe drive shows up under “PCIE Drives” > “MiniSAS PCIe SSD-0”.

      Let me know what data you need and I will get it up.

      george1421G 1 Reply Last reply Reply Quote 0
      • george1421G
        george1421 Moderator @hlalex
        last edited by

        @hlalex ok next question, since you are using a dell, and can we assume that you are pxe booting into uefi mode? If so, go back into the firmware and change the disk mode from “Raid-on” to “AHCI” mode. Then pxe boot your computer. There is a documented issue with LINUX and the intel rapidstore hardware when raid-on is enabled. After imaging but before the first OS boot you can switch it back with no harm. Or just leave it in ahci mode and have no harm.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        1 Reply Last reply Reply Quote 0
        • H
          hlalex
          last edited by

          We actually use Legacy mode. We have found it to be much easier to manage, especially with our dual-boot images (W10 + Fedora 27).

          Switching to “AHCI” is one of our first changes before attempting to push an image. We also enable Legacy option ROMs and set the boot sequence to Legacy.

          We are able to image the Hynix drives in the new 5820 systems after removing the M.2 from the 5820 FlexBay port and installing it in a 5810 PCIe adapter. This makes us think the hickup is related to the pxe/tftpboot/bzImage process detecting the drive due to some difference introduced by the MiniSAS/backplane.

          Our typical BIOS changes:
          Enable Legacy Option ROMs
          Set boot option to Legacy
          SATA Operation = AHCI
          Disable Intel VMD Technology
          TPM off
          Secure Boot Disabled

          1 Reply Last reply Reply Quote 0
          • george1421G
            george1421 Moderator
            last edited by

            OK well, lets put everything back in the system and then schedule a debug deploy (tick the debug checkbox when you setup a deploy task). Then pxe boot the target computer. After a few key presses FOS will drop you to a command prompt on the target computer.

            These next steps will make it easier for you to copy and paste in FOS.

            1. Get the IP address of the FOS target computer with ip addr show. Record this value for future reference.
            2. Give root a temporary password in FOS. Key in passwd and give root a simple password like hello. I might complain, but it will take the simple password.
            3. Now use the free ssh program called putty to connect to the FOS engine using the IP address from step 1. Login as root with the password of hello from step 2.

            Now from here you should have console access in the FOS engine. From here lets collect a bit of information.

            1. First we will collect what devices are found in this system. Copy the results of this command into a reply in this thread:
              lspci -nn
            2. Lets collect the block devices FOS see with:
              lsblk

            We may need to review the /var/syslog but for now lets see what is returned with these commands.

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

            1 Reply Last reply Reply Quote 0
            • H
              hlalex
              last edited by

              Ok with the system back in the original hardware configuration (BIOS settings as listed in my last comment):

              lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:2020] (rev 04)
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
00:05.2 System peripheral [0880]: Intel Corporation Device [8086:2025] (rev 04)
00:05.4 PIC [0800]: Intel Corporation Device [8086:2026] (rev 04)
00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
00:14.0 USB controller [0c03]: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller [8086:a2af]
00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
00:17.0 RAID bus controller [0104]: Intel Corporation C600/X79 series chipset SATA RAID Controller [8086:2826]
00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
00:1c.6 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #7 [8086:a296] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2d3]
00:1f.2 Memory controller [0580]: Intel Corporation 200 Series PCH PMC [8086:a2a1]
00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
00:1f.4 SMBus [0c05]: Intel Corporation 200 Series PCH SMBus Controller [8086:a2a3]
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (5) I219-LM [8086:15e3]
02:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
16:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
16:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
64:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
64:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
64:08.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
64:09.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
64:0a.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
64:0a.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
64:0a.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
64:0a.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
64:0a.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
64:0a.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
64:0a.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
64:0a.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
64:0b.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
64:0b.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
64:0b.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
64:0b.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
64:0c.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
64:0c.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
64:0c.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
64:0c.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
64:0c.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
64:0c.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
64:0c.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
64:0c.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
64:0d.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
64:0d.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
64:0d.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
64:0d.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
65:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2000] [10de:1c30] (rev a1)
65:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
b2:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 04)
b2:03.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port D [8086:2033] (rev 04)
b2:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
b2:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)

              lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb      8:16   1  3.8G  0 disk 
`-sdb1   8:17   1    1M  0 part

              Due to the network configuration I have to use a custom iPXE rom, the 3.8G /dev/sdb that shows up is the USB drive with the iPXE bootloader

              george1421G 1 Reply Last reply Reply Quote 0
              • george1421G
                george1421 Moderator @hlalex
                last edited by

                @hlalex Well we know a few things now.

                The only sata controller I see is:

                00:17.0 RAID bus controller [0104]: Intel Corporation C600/X79 series chipset SATA RAID Controller [8086:2826]

                But lsblk doesn’t see any drives in that system. On the host configuration for this system, add the following kernel parameter and reboot back into the debug deploy.
                mdraid=true

                I have a tutorial on how to configure FOS to see an intel raid configuraiton: https://forums.fogproject.org/topic/7882/capture-deploy-to-target-computers-using-intel-rapid-storage-onboard-raid

                I seriously don’t think this will solve your issue because we can’t see any existing disks at the moment.

                Since you have fedora working on this system, maybe we can use that to reverse engineer how Fedora make it work.

                Boot into fedora and at the linux command prompt key in
                sudo lspci -k -nn and post the output here. This will show the pci device with the kernel driver that is controlling it.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                1 Reply Last reply Reply Quote 0
                • H
                  hlalex
                  last edited by hlalex

                  EDIT: This was from a Fedora 27 Live CD.

                  I think the very last entry is what we are looking for.

                  sudo lspci -k -nn
00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:05.2 System peripheral [0880]: Intel Corporation Device [8086:2025] (rev 04)
00:05.4 PIC [0800]: Intel Corporation Device [8086:2026] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:14.0 USB controller [0c03]: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller [8086:a2af]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: xhci_hcd
00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
	Subsystem: Dell Device [1028:0738]
00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: mei_me
	Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1c.6 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #7 [8086:a296] (rev f0)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2d3]
	Subsystem: Dell Device [1028:0738]
00:1f.2 Memory controller [0580]: Intel Corporation 200 Series PCH PMC [8086:a2a1]
	Subsystem: Dell Device [1028:0738]
00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
00:1f.4 SMBus [0c05]: Intel Corporation 200 Series PCH SMBus Controller [8086:a2a3]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (5) I219-LM [8086:15e3]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: e1000e
	Kernel modules: e1000e
02:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
	Kernel modules: shpchp
16:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
16:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
64:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
64:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:08.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:09.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0a.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0a.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0a.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0b.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0b.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0b.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0b.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0c.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0c.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0d.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0d.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0d.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0d.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
	Subsystem: Dell Device [1028:0738]
65:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2000] [10de:1c30] (rev a1)
	Subsystem: Dell Device [1028:11b3]
	Kernel driver in use: nouveau
	Kernel modules: nouveau
65:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
	Subsystem: Dell Device [1028:11b3]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
b2:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 04)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
b2:03.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port D [8086:2033] (rev 04)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
b2:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
b2:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]
	Subsystem: Device [1c5c:1527]
	Kernel driver in use: nvme
	Kernel modules: nvme

                  Just for completeness:

                  lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0   1.4G  1 loop 
loop1         7:1    0   6.5G  1 loop 
├─live-rw   253:0    0   6.5G  0 dm   /
└─live-base 253:1    0   6.5G  1 dm   
loop2         7:2    0    32G  0 loop 
└─live-rw   253:0    0   6.5G  0 dm   /
sdb           8:16   1  14.6G  0 disk 
└─sdb1        8:17   1  14.6G  0 part /run/initramfs/live
sr0          11:0    1  1024M  0 rom  
nvme0n1     259:0    0   477G  0 disk 
├─nvme0n1p1 259:1    0   650M  0 part 
├─nvme0n1p2 259:2    0   128M  0 part 
├─nvme0n1p3 259:3    0   499M  0 part 
└─nvme0n1p4 259:4    0 475.7G  0 part
                  george1421G 1 Reply Last reply Reply Quote 0
                  • george1421G
                    george1421 Moderator @hlalex
                    last edited by

                    @hlalex I think you are spot on with your kernel driver name. The lsblk command from fedora is what I should expect from FOS.

                    One last thing I should have asked for with fedora, will you provide the output of sudo uname -a That will tell us what kernel they are using.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                    1 Reply Last reply Reply Quote 0
                    • H
                      hlalex
                      last edited by 

                      Linux localhost-live 4.13.9-300.fc27.x86_64 #1 SMP Mon Oct 23 13:41:58 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
                      george1421G 1 Reply Last reply Reply Quote 0
                      • george1421G
                        george1421 Moderator @hlalex
                        last edited by

                        @hlalex Well after some research, I think I found what is missing. We may need to get the developers involved to recompile the FOS kernel.

                        First some background.
                        FC27
                        Kernel used 4.13.9
                        Device discovered

                        b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]
	Subsystem: Device [1c5c:1527]
	Kernel driver in use: nvme
	Kernel modules: nvme

                        FOS 1.5.4
                        Kernel used 4.17.0 (or 4.15.2)
                        Device b3:00.0 not found/kernel driver not available.

                        Linux kernel config file found here: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config

                        Setting disabled is on line 2925: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config#L2925

                        #  CONFIG_NVMEM is not set

                        CONFIG_NVMEM == "Support for NVMEM(Non Volatile Memory) devices like EEPROM, EFUSES…

                        This framework is designed to provide a generic interface to NVMEM from both the Linux Kernel and the userspace.

                        This driver can also be built as a module. If so, the module will be called nvmem_core."

                        @Developers Opinion here? Is there a possibility to get this option turned on for linux kernels 4.15.2 and 4.17.0 for testing purposes?

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                        1 Reply Last reply Reply Quote 0
                        • H
                          hlalex
                          last edited by

                          I just received word from Dell that they are going to swap the FlexBay Carriers out for standard PCIe adapters (similar model to the adapters in the Precision 5810s) that should hopefully work without issues.
                          I would like to provide as much information as I can before I have to ship them out, so please let me know what other logs/data/tests we may need to work out the issue. I’m sure these drives will become more prevalent in the near future and would like to help as much as I can. Not to mention I have another 40 system lab to refresh next summer, and I would love to have a solution if they wind up with this same drive configuration.

                          george1421G 1 Reply Last reply Reply Quote 0
                          • george1421G
                            george1421 Moderator @hlalex
                            last edited by

                            @hlalex We surely like to get this addressed before you swap out the hardware. I’m sure others will either order the same hardware as you have, or other manufacturers will use this hardware design. It needs to be fixed.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                            1 Reply Last reply Reply Quote 0
                            • H
                              hlalex
                              last edited by

                              @george1421 Agreed. It looks like the replacements will not ship until 8/14 so we have a bit of time for testing.

                              What baffles me is that I have been in contact with multiple Dell Reps trying to determine exactly which hardware needs to be returned, and none have been able to answer what part is what or point me to any product information for this drive type. As far as I can tell from their site, this particular drive configuration is not currently offered as a retail addition. The specs related to the drive based on the dell support Service Tag lookup lists only:

                              414-BBBV : PCIe SSD (Front PCIe FlexBay)
                              Part Number Quantity Description
                              5G90D 1 INFORMATION, FRONT, PERIPHERAL COMPONENT INTERCONNECT EXPRESS , SOFTWARE SUPPORT DISKETTE, BOOT

                              401-ABJT : M.2 512GB PCIe NVMe Class 40 S olid State Drive
                              Part Number Quantity Description
                              XMW6J 1 SSDR, 512G, P34, 80S3, HYNIX, PC401

                              400-AVDR : Dell M.2 carrier
                              Part Number Quantity Description
                              66XHV 1 ASSEMBLY, DRIVE, BAY (DRIVE BAY), M.2, MODULE

                              I have looked high and low for any information on these part numbers with no luck.

                              george1421G 1 Reply Last reply Reply Quote 0
                              • george1421G
                                george1421 Moderator @hlalex
                                last edited by

                                @hlalex I went ahead and compiled the latest FOS kernel with the suspected missing module enabled. This is kernel version 4.17.13. Understand its not the official FOS kernel because it doesn’t have the customized patches that FOG kernel developers add, but should work for our test to see if we can init that nvme memory device.

                                I’ll IM you a link to the test kernel.

                                For this test download the linked kernel to /var/www/html/fog/service/ipxe on the fog server, leave the name as bzImage41713 Then go into the host record for one of these test systems and set the “Host Kernel” parameter to bzImage41713. Then finally pxe boot the target system into debug capture task. Run the lspci -nn command and see if we can detect

                                b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]

                                If its in the list then run the lsblk command to see if has something we can mount.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                1 Reply Last reply Reply Quote 0
                                • H
                                  hlalex
                                  last edited by

                                  Good Morning @george1421. I stopped by the office to test the custom kernel (I’m going to pick up breakfast for my wife on my way home so she should be happy ;). No luck. The first attempt to boot gave init error, and I realized it was trying init_32.xz instead of init.xz. After adding init.xz to the "Host Init " parameter in the host record it booted into debug without issue. Bad news is that

                                  b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]

                                  does not show up with lspci -nn and only /dev/sdb (pxe boot usb) shows up with lsblk.

                                  george1421G 1 Reply Last reply Reply Quote 0
                                  • george1421G
                                    george1421 Moderator @hlalex
                                    last edited by

                                    @hlalex Well, digging a bit deeper into this…

                                    Going line by line…

                                    FC27 I see that the disk controller is in SATA mode

                                    00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ahci

                                    FOS the disk controllers was in Raid-on mode

                                    00:17.0 RAID bus controller [0104]: Intel Corporation C600/X79 series chipset SATA RAID Controller [8086:2826]

                                    Note that the hardware ID changes between the two modes. (not pointing fingers here) But as I said before linux does not see the disks behind raid-on devices in uefi mode. Even if you are in bios mode, please change the value to ahci mode and pull the lspci from FOS. I don’t believe that lspci under FOS supports the -k prompt, but that info would be handy.

                                    I also found another kernel module that was disabled called “Devices/Memory Controller drivers”. After a bit more research I’m going to compile and updated kernel with Memory Controller support enabled.

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                    1 Reply Last reply Reply Quote 0
                                    • H
                                      hlalex
                                      last edited by

                                      @george1421 I had noticed that as well, and double checked the Raid settings. It had been turned back on at some point (probably while I was going through everything else). I turned it back off and re-did the debug task with the same results.

                                      It appears that the NVMe PCIe drives use an entirely separate controller from SATA drives. Apparently the specific controller/driver varies based on the interface between the MoBo and SSD. FOS can see the M.2 drives when the are connected via a standard PCIe adapter card, however when the same M.2 drive is connected via the FlexBay MiniSAS backplane, FOS no longer recognizes the drive. This holds true regardless of the SATA controller configuration–the drives are visible with both Raid & AHCI modes.

                                      I found a dell support article that gives some good information about the differences between NVMe and AHCI here.

                                      I checked through the FOS kernel, and found NVME Support is enabled in lines 880-890: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config#L880

                                      #
# NVME Support
#
CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_FABRICS=y
CONFIG_NVME_FC=y
CONFIG_NVME_TARGET=y
CONFIG_NVME_TARGET_LOOP=y
CONFIG_NVME_TARGET_FC=y
CONFIG_NVME_TARGET_FCLOOP=y

                                      Which explains why the drives are detected when plugged into a PCIe adapter. The MiniSAS connection is a completely different beast–or at least uses a different controller from the standard NVMe drive.

                                      I will run through the debug again (with Raid off in bios) and post the results of lspci -nn, lsblk, uname -a, and anything else you suggest. I will also do the same thing with the M.2 plugged in through a PCIe adapter so we have comparison data.

                                      george1421G 1 Reply Last reply Reply Quote 0
                                      • george1421G
                                        george1421 Moderator @hlalex
                                        last edited by george1421

                                        @hlalex I have 2 additional kernels for you to test, but I will only share with you tomorrow since you should have other activities planned for today.

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                        H 1 Reply Last reply Reply Quote 0
                                        • H
                                          hlalex @george1421
                                          last edited by

                                          @george1421 Ok, I scripted data collection and pulled fresh info for kernels a, b, c, and F27 Live. I also pulled dmidecode, full lshw ouput, and /proc/diskstats to additional log files. Let me know if they may be of any use I will upload them as well.

                                          bzImage41713a.log
                                          bzImage41713b.log
                                          bzImage41713c.log
                                          F27_4139300.log

                                          george1421G 2 Replies Last reply Reply Quote 1
                                          • george1421G
                                            george1421 Moderator @hlalex
                                            last edited by george1421

                                            @hlalex Very nice. I see somethings in the FC syslog, that are missing in FOS, namely PCI Hotplug and FOS is not initing all of the CPUs, FOS stops at 8. The Developers may need to reconsider an upper limit with these new systems.

                                            That “memory controller” is surely the device we are after.

                                            b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]

# and
[0.755229] pci 0000:b3:00.0: [1c5c:1527] type 00 class 0x010802
[0.755243] pci 0000:b3:00.0: reg 0x10: [mem 0xfb500000-0xfb503fff 64bit]

# and

[4.586882] nvme nvme0: pci function 0000:b3:00.0
[4.803854] nvme0n1: p1 p2 p3 p4

                                            I’m going to look into a few of the messages I diff’d between the two syslogs and see what I can tweak in the kernel configuration.

                                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                            1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 5
                                            • 6
                                            • 7
                                            • 7 / 7
                                            • First post
                                              Last post

                                            200

                                            Online

                                            11.6k

                                            Users

                                            17.1k

                                            Topics

                                            154.6k

                                            Posts
                                            Copyright © 2012-2024 FOG Project