• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized

    Scheduled Pinned Locked Moved
    Hardware Compatibility
    9
    124
    38.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H
      hlalex
      last edited by

      First, I apologize if this is a duplicate thread. I have searched for days and not found anything that resembles our current issue. We have been using FOG for several years to image approximately 100 computers in 5 different labs. Up to this point we have not encountered any major problems that we could not find a documented solution for.

      Our most recent refresh presented us with Dell’s Flex Bay MiniSAS PCIe NVME SSDs. Fog does not recognize the drives when in the Flex Bay MiniSAS adapter, however when we swap the M.2 SSDs over to a standard PCIe adapter, Fog has no issues recognizing the drives. We are running FOG Version 1.5.4 and have tried the default Kernel and updated Kernels 4.17.0-addbnx 64 and 4.17.0 64 with no luck.

      We have also attempted booting using Clonezilla Live 2.5.5-38, and received error: “nvme nvme0n: failed to set APST feature (-19)”. Per This Bug, we have tested several values for “nvme_core.default_ps_max_latency_us=” with no luck. The only unix bootable ISO that has successfully detected the drive is Fedora 27 live CD.

      We are contemplating dissecting the Fedora Kernel and Grub patches to build a custom bzImage, but wanted to reach out to the community to see if anyone had alternate suggestions first.

      More system information:
      Dell Precision Tower 5820
      512GB Hynix M.2 PCIe NVMe Class 40 SSD
      Front PCIe FlexBay (Shows up as MiniSAS PCIe SSD-0 in BIOS)
      Intel Integrated controller (R ST-e) with 1-2 Front FlexBay NVMe PCIe Drives

      1 Reply Last reply Reply Quote 0
      • george1421G
        george1421 Moderator
        last edited by

        Do you have your storage setup as a raid?

        If not we may need to do some data collection steps to find out a bit more about that hardware.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        1 Reply Last reply Reply Quote 0
        • H
          hlalex
          last edited by

          Hey george,

          We do not have raid on the FOG server storage, and as far as I can tell the 5820s do not either. I have tried [CTRL] + I during boot and nothing comes up. Also, in BIOS the only raid options I see pertain to SATA drives (also disabled), while the NVMe drive shows up under “PCIE Drives” > “MiniSAS PCIe SSD-0”.

          Let me know what data you need and I will get it up.

          george1421G 1 Reply Last reply Reply Quote 0
          • george1421G
            george1421 Moderator @hlalex
            last edited by

            @hlalex ok next question, since you are using a dell, and can we assume that you are pxe booting into uefi mode? If so, go back into the firmware and change the disk mode from “Raid-on” to “AHCI” mode. Then pxe boot your computer. There is a documented issue with LINUX and the intel rapidstore hardware when raid-on is enabled. After imaging but before the first OS boot you can switch it back with no harm. Or just leave it in ahci mode and have no harm.

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

            1 Reply Last reply Reply Quote 0
            • H
              hlalex
              last edited by

              We actually use Legacy mode. We have found it to be much easier to manage, especially with our dual-boot images (W10 + Fedora 27).

              Switching to “AHCI” is one of our first changes before attempting to push an image. We also enable Legacy option ROMs and set the boot sequence to Legacy.

              We are able to image the Hynix drives in the new 5820 systems after removing the M.2 from the 5820 FlexBay port and installing it in a 5810 PCIe adapter. This makes us think the hickup is related to the pxe/tftpboot/bzImage process detecting the drive due to some difference introduced by the MiniSAS/backplane.

              Our typical BIOS changes:
              Enable Legacy Option ROMs
              Set boot option to Legacy
              SATA Operation = AHCI
              Disable Intel VMD Technology
              TPM off
              Secure Boot Disabled

              1 Reply Last reply Reply Quote 0
              • george1421G
                george1421 Moderator
                last edited by

                OK well, lets put everything back in the system and then schedule a debug deploy (tick the debug checkbox when you setup a deploy task). Then pxe boot the target computer. After a few key presses FOS will drop you to a command prompt on the target computer.

                These next steps will make it easier for you to copy and paste in FOS.

                1. Get the IP address of the FOS target computer with ip addr show. Record this value for future reference.
                2. Give root a temporary password in FOS. Key in passwd and give root a simple password like hello. I might complain, but it will take the simple password.
                3. Now use the free ssh program called putty to connect to the FOS engine using the IP address from step 1. Login as root with the password of hello from step 2.

                Now from here you should have console access in the FOS engine. From here lets collect a bit of information.

                1. First we will collect what devices are found in this system. Copy the results of this command into a reply in this thread:
                  lspci -nn
                2. Lets collect the block devices FOS see with:
                  lsblk

                We may need to review the /var/syslog but for now lets see what is returned with these commands.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                1 Reply Last reply Reply Quote 0
                • H
                  hlalex
                  last edited by

                  Ok with the system back in the original hardware configuration (BIOS settings as listed in my last comment):

                  lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:2020] (rev 04)
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
00:05.2 System peripheral [0880]: Intel Corporation Device [8086:2025] (rev 04)
00:05.4 PIC [0800]: Intel Corporation Device [8086:2026] (rev 04)
00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
00:14.0 USB controller [0c03]: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller [8086:a2af]
00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
00:17.0 RAID bus controller [0104]: Intel Corporation C600/X79 series chipset SATA RAID Controller [8086:2826]
00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
00:1c.6 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #7 [8086:a296] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2d3]
00:1f.2 Memory controller [0580]: Intel Corporation 200 Series PCH PMC [8086:a2a1]
00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
00:1f.4 SMBus [0c05]: Intel Corporation 200 Series PCH SMBus Controller [8086:a2a3]
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (5) I219-LM [8086:15e3]
02:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
16:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
16:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
64:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
64:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
64:08.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
64:09.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
64:0a.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
64:0a.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
64:0a.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
64:0a.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
64:0a.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
64:0a.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
64:0a.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
64:0a.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
64:0b.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
64:0b.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
64:0b.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
64:0b.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
64:0c.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
64:0c.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
64:0c.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
64:0c.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
64:0c.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
64:0c.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
64:0c.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
64:0c.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
64:0d.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
64:0d.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
64:0d.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
64:0d.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
65:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2000] [10de:1c30] (rev a1)
65:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
b2:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 04)
b2:03.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port D [8086:2033] (rev 04)
b2:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
b2:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)

                  lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb      8:16   1  3.8G  0 disk 
`-sdb1   8:17   1    1M  0 part

                  Due to the network configuration I have to use a custom iPXE rom, the 3.8G /dev/sdb that shows up is the USB drive with the iPXE bootloader

                  george1421G 1 Reply Last reply Reply Quote 0
                  • george1421G
                    george1421 Moderator @hlalex
                    last edited by

                    @hlalex Well we know a few things now.

                    The only sata controller I see is:

                    00:17.0 RAID bus controller [0104]: Intel Corporation C600/X79 series chipset SATA RAID Controller [8086:2826]

                    But lsblk doesn’t see any drives in that system. On the host configuration for this system, add the following kernel parameter and reboot back into the debug deploy.
                    mdraid=true

                    I have a tutorial on how to configure FOS to see an intel raid configuraiton: https://forums.fogproject.org/topic/7882/capture-deploy-to-target-computers-using-intel-rapid-storage-onboard-raid

                    I seriously don’t think this will solve your issue because we can’t see any existing disks at the moment.

                    Since you have fedora working on this system, maybe we can use that to reverse engineer how Fedora make it work.

                    Boot into fedora and at the linux command prompt key in
                    sudo lspci -k -nn and post the output here. This will show the pci device with the kernel driver that is controlling it.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                    1 Reply Last reply Reply Quote 0
                    • H
                      hlalex
                      last edited by hlalex

                      EDIT: This was from a Fedora 27 Live CD.

                      I think the very last entry is what we are looking for.

                      sudo lspci -k -nn
00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma
00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:05.2 System peripheral [0880]: Intel Corporation Device [8086:2025] (rev 04)
00:05.4 PIC [0800]: Intel Corporation Device [8086:2026] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
	Subsystem: Dell Device [1028:0738]
00:14.0 USB controller [0c03]: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller [8086:a2af]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: xhci_hcd
00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
	Subsystem: Dell Device [1028:0738]
00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: mei_me
	Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1c.6 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #7 [8086:a296] (rev f0)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2d3]
	Subsystem: Dell Device [1028:0738]
00:1f.2 Memory controller [0580]: Intel Corporation 200 Series PCH PMC [8086:a2a1]
	Subsystem: Dell Device [1028:0738]
00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
00:1f.4 SMBus [0c05]: Intel Corporation 200 Series PCH SMBus Controller [8086:a2a3]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (5) I219-LM [8086:15e3]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: e1000e
	Kernel modules: e1000e
02:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
	Kernel modules: shpchp
16:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
16:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
	Subsystem: Dell Device [1028:0738]
16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
64:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
64:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:08.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:09.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0a.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0a.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0a.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0a.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0b.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0b.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0b.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0b.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0c.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0c.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0c.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0d.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0d.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
	Subsystem: Dell Device [1028:0738]
64:0d.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
64:0d.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
	Subsystem: Dell Device [1028:0738]
65:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2000] [10de:1c30] (rev a1)
	Subsystem: Dell Device [1028:11b3]
	Kernel driver in use: nouveau
	Kernel modules: nouveau
65:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
	Subsystem: Dell Device [1028:11b3]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
b2:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 04)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
b2:03.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port D [8086:2033] (rev 04)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
b2:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
b2:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: skx_uncore
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
	Subsystem: Dell Device [1028:0738]
b2:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]
	Subsystem: Device [1c5c:1527]
	Kernel driver in use: nvme
	Kernel modules: nvme

                      Just for completeness:

                      lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0   1.4G  1 loop 
loop1         7:1    0   6.5G  1 loop 
├─live-rw   253:0    0   6.5G  0 dm   /
└─live-base 253:1    0   6.5G  1 dm   
loop2         7:2    0    32G  0 loop 
└─live-rw   253:0    0   6.5G  0 dm   /
sdb           8:16   1  14.6G  0 disk 
└─sdb1        8:17   1  14.6G  0 part /run/initramfs/live
sr0          11:0    1  1024M  0 rom  
nvme0n1     259:0    0   477G  0 disk 
├─nvme0n1p1 259:1    0   650M  0 part 
├─nvme0n1p2 259:2    0   128M  0 part 
├─nvme0n1p3 259:3    0   499M  0 part 
└─nvme0n1p4 259:4    0 475.7G  0 part
                      george1421G 1 Reply Last reply Reply Quote 0
                      • george1421G
                        george1421 Moderator @hlalex
                        last edited by

                        @hlalex I think you are spot on with your kernel driver name. The lsblk command from fedora is what I should expect from FOS.

                        One last thing I should have asked for with fedora, will you provide the output of sudo uname -a That will tell us what kernel they are using.

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                        1 Reply Last reply Reply Quote 0
                        • H
                          hlalex
                          last edited by 

                          Linux localhost-live 4.13.9-300.fc27.x86_64 #1 SMP Mon Oct 23 13:41:58 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
                          george1421G 1 Reply Last reply Reply Quote 0
                          • george1421G
                            george1421 Moderator @hlalex
                            last edited by

                            @hlalex Well after some research, I think I found what is missing. We may need to get the developers involved to recompile the FOS kernel.

                            First some background.
                            FC27
                            Kernel used 4.13.9
                            Device discovered

                            b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]
	Subsystem: Device [1c5c:1527]
	Kernel driver in use: nvme
	Kernel modules: nvme

                            FOS 1.5.4
                            Kernel used 4.17.0 (or 4.15.2)
                            Device b3:00.0 not found/kernel driver not available.

                            Linux kernel config file found here: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config

                            Setting disabled is on line 2925: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config#L2925

                            #  CONFIG_NVMEM is not set

                            CONFIG_NVMEM == "Support for NVMEM(Non Volatile Memory) devices like EEPROM, EFUSES…

                            This framework is designed to provide a generic interface to NVMEM from both the Linux Kernel and the userspace.

                            This driver can also be built as a module. If so, the module will be called nvmem_core."

                            @Developers Opinion here? Is there a possibility to get this option turned on for linux kernels 4.15.2 and 4.17.0 for testing purposes?

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                            1 Reply Last reply Reply Quote 0
                            • H
                              hlalex
                              last edited by

                              I just received word from Dell that they are going to swap the FlexBay Carriers out for standard PCIe adapters (similar model to the adapters in the Precision 5810s) that should hopefully work without issues.
                              I would like to provide as much information as I can before I have to ship them out, so please let me know what other logs/data/tests we may need to work out the issue. I’m sure these drives will become more prevalent in the near future and would like to help as much as I can. Not to mention I have another 40 system lab to refresh next summer, and I would love to have a solution if they wind up with this same drive configuration.

                              george1421G 1 Reply Last reply Reply Quote 0
                              • george1421G
                                george1421 Moderator @hlalex
                                last edited by

                                @hlalex We surely like to get this addressed before you swap out the hardware. I’m sure others will either order the same hardware as you have, or other manufacturers will use this hardware design. It needs to be fixed.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                1 Reply Last reply Reply Quote 0
                                • H
                                  hlalex
                                  last edited by

                                  @george1421 Agreed. It looks like the replacements will not ship until 8/14 so we have a bit of time for testing.

                                  What baffles me is that I have been in contact with multiple Dell Reps trying to determine exactly which hardware needs to be returned, and none have been able to answer what part is what or point me to any product information for this drive type. As far as I can tell from their site, this particular drive configuration is not currently offered as a retail addition. The specs related to the drive based on the dell support Service Tag lookup lists only:

                                  414-BBBV : PCIe SSD (Front PCIe FlexBay)
                                  Part Number Quantity Description
                                  5G90D 1 INFORMATION, FRONT, PERIPHERAL COMPONENT INTERCONNECT EXPRESS , SOFTWARE SUPPORT DISKETTE, BOOT

                                  401-ABJT : M.2 512GB PCIe NVMe Class 40 S olid State Drive
                                  Part Number Quantity Description
                                  XMW6J 1 SSDR, 512G, P34, 80S3, HYNIX, PC401

                                  400-AVDR : Dell M.2 carrier
                                  Part Number Quantity Description
                                  66XHV 1 ASSEMBLY, DRIVE, BAY (DRIVE BAY), M.2, MODULE

                                  I have looked high and low for any information on these part numbers with no luck.

                                  george1421G 1 Reply Last reply Reply Quote 0
                                  • george1421G
                                    george1421 Moderator @hlalex
                                    last edited by

                                    @hlalex I went ahead and compiled the latest FOS kernel with the suspected missing module enabled. This is kernel version 4.17.13. Understand its not the official FOS kernel because it doesn’t have the customized patches that FOG kernel developers add, but should work for our test to see if we can init that nvme memory device.

                                    I’ll IM you a link to the test kernel.

                                    For this test download the linked kernel to /var/www/html/fog/service/ipxe on the fog server, leave the name as bzImage41713 Then go into the host record for one of these test systems and set the “Host Kernel” parameter to bzImage41713. Then finally pxe boot the target system into debug capture task. Run the lspci -nn command and see if we can detect

                                    b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]

                                    If its in the list then run the lsblk command to see if has something we can mount.

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                    1 Reply Last reply Reply Quote 0
                                    • H
                                      hlalex
                                      last edited by

                                      Good Morning @george1421. I stopped by the office to test the custom kernel (I’m going to pick up breakfast for my wife on my way home so she should be happy ;). No luck. The first attempt to boot gave init error, and I realized it was trying init_32.xz instead of init.xz. After adding init.xz to the "Host Init " parameter in the host record it booted into debug without issue. Bad news is that

                                      b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]

                                      does not show up with lspci -nn and only /dev/sdb (pxe boot usb) shows up with lsblk.

                                      george1421G 1 Reply Last reply Reply Quote 0
                                      • george1421G
                                        george1421 Moderator @hlalex
                                        last edited by

                                        @hlalex Well, digging a bit deeper into this…

                                        Going line by line…

                                        FC27 I see that the disk controller is in SATA mode

                                        00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
	Subsystem: Dell Device [1028:0738]
	Kernel driver in use: ahci

                                        FOS the disk controllers was in Raid-on mode

                                        00:17.0 RAID bus controller [0104]: Intel Corporation C600/X79 series chipset SATA RAID Controller [8086:2826]

                                        Note that the hardware ID changes between the two modes. (not pointing fingers here) But as I said before linux does not see the disks behind raid-on devices in uefi mode. Even if you are in bios mode, please change the value to ahci mode and pull the lspci from FOS. I don’t believe that lspci under FOS supports the -k prompt, but that info would be handy.

                                        I also found another kernel module that was disabled called “Devices/Memory Controller drivers”. After a bit more research I’m going to compile and updated kernel with Memory Controller support enabled.

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                        1 Reply Last reply Reply Quote 0
                                        • H
                                          hlalex
                                          last edited by

                                          @george1421 I had noticed that as well, and double checked the Raid settings. It had been turned back on at some point (probably while I was going through everything else). I turned it back off and re-did the debug task with the same results.

                                          It appears that the NVMe PCIe drives use an entirely separate controller from SATA drives. Apparently the specific controller/driver varies based on the interface between the MoBo and SSD. FOS can see the M.2 drives when the are connected via a standard PCIe adapter card, however when the same M.2 drive is connected via the FlexBay MiniSAS backplane, FOS no longer recognizes the drive. This holds true regardless of the SATA controller configuration–the drives are visible with both Raid & AHCI modes.

                                          I found a dell support article that gives some good information about the differences between NVMe and AHCI here.

                                          I checked through the FOS kernel, and found NVME Support is enabled in lines 880-890: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config#L880

                                          #
# NVME Support
#
CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_FABRICS=y
CONFIG_NVME_FC=y
CONFIG_NVME_TARGET=y
CONFIG_NVME_TARGET_LOOP=y
CONFIG_NVME_TARGET_FC=y
CONFIG_NVME_TARGET_FCLOOP=y

                                          Which explains why the drives are detected when plugged into a PCIe adapter. The MiniSAS connection is a completely different beast–or at least uses a different controller from the standard NVMe drive.

                                          I will run through the debug again (with Raid off in bios) and post the results of lspci -nn, lsblk, uname -a, and anything else you suggest. I will also do the same thing with the M.2 plugged in through a PCIe adapter so we have comparison data.

                                          george1421G 1 Reply Last reply Reply Quote 0
                                          • george1421G
                                            george1421 Moderator @hlalex
                                            last edited by george1421

                                            @hlalex I have 2 additional kernels for you to test, but I will only share with you tomorrow since you should have other activities planned for today.

                                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                            H 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 5
                                            • 6
                                            • 7
                                            • 1 / 7
                                            • First post
                                              Last post

                                            174

                                            Online

                                            11.6k

                                            Users

                                            17.1k

                                            Topics

                                            154.6k

                                            Posts
                                            Copyright © 2012-2024 FOG Project