Posts made by hlalex

@george1421 version L logs:

• bzImage41713l.log
• bzImage41713l_lshw.log

@george1421 That’s a great addition, especially with all the new devices with USB-C (these 5820s have 2 front C ports).
Here are logs from Rev K:

• bzImage41713k.log
• bzImage41713k_lshw.log

@george1421 rev J boots just fine, but I did notice an error message to the effect of “/dev … error creating epoll fd”. It was gone almost before I saw it, but that is what I was able to remember. I vaguely remember seeing this error before, but I do not know if it was with any of the other test kernels or some other project.

Here are the logs:

• bzImage41713j.log
• bzImage41713j_lshw.log

@george1421 No worries! I run into gremlins like that all the time. I’ll keep an eye out for updates.

@george1421 rev I logs:

• bzImage41713i.log
• bzImage41713i_lshw.log

@george1421 If you would like I can upload the bash script I am using to collect the data. Its very basic, but it speeds things up quite a bit (and automatically masks personal info).

@george1421 Same here, The 5820 bios has some different options regarding NVMe drives connected to the FrontFlex Bay. These devices actually show up in bios, while the same drive connected via PCIe adapter does not populate under any bios menu that I have located. This suggest something is very different at the bios level for the two connection methods.

version C

• Boots
• Error parsing PCC subspaces from PCCT
• NVMe not in lsblk

version D

• Boots
• Error parsing PCC subspaces from PCCT
• nvme nvme0: failed to set APST feature (-19)
• NVMe not in lsblk

version E

• Kernel Panic
• Error parsing PCC subspaces from PCCT
• pciehp 0000:b2:02.0:pcie004: Slot(12): Power Fault
• pciehp 0000:b2:03.0:pcie004: Slot(13): Power Fault
• acpiphp_ibm: ibm_acpiphp_init: acpi_walk_namespace failed
• nvme nvme0: failed to set APST feature (-19)
• (/sbin/init & /bin/sh) exists but couldn’t execute it (error -8)

version F

• kernel panic
• Error parsing PCC subspaces from PCCT
• acpiphp_ibm: ibm_acpiphp_init: acpi_walk_namespace failed error
• (/sbin/init & /bin/sh) exists but couldn’t execute it (error -8)

version G

• Boots (No idea what I was doing wrong before…)
• Error parsing PCC subspaces from PCCT
• bzImage41713g.log
• bzImage41713g_lsmod.log

@george1421 Ok, double checked everything and it appears the bios change didn’t have any effect on the output of any kernels. Same errors and freezes.

@george1421 Sorry, got caught up with some other issues. I tried G, but it hangs at

Configuring (net0 xx:xx:xx:xx:xx:xx)... ok
http://ip.addr/fog/service/ipxe/boot.php... ok
bzImage41713g... ok
init.xz... ok
_

I have double checked and init.xz is still set as default.

Hang on, just found an issue with the BIOS. Testing D - G again to see if it had any effect.

@george1421 F produced a similar kernel panic:

@george1421 Didn’t touch it at all. I thought about changing it to init_32.xz, but decided to wait for you.

@george1421 Not a problem! FOG has saved me countless hours of manual configuration over the years, and until this recent batch of hardware I have not come across an issue that wasn’t already answered/fixed in another post. I am happy to help in any way I can, and hopefully save someone else (and possibly my future self) a lot of headache. We got extremely lucky with the unequal exchange approval from Dell, and other techs may not get the same break (It took 2 days and many hours on the phone/sending emails to get the exchange).

@george1421 No luck on version E. We got the APST feature error, however FOS failed with a kernel panic.

@george1421 We are making some progress with version D. While booting I received the

print_req_error: I/O error, dev nvme0n1, sector 0
nvme nvme0: failed to set APST feature (-19)
unable to open '/dev/nvme0n1'
syspath not found

error I was getting with Clonezilla live. This is the first time this error has shown up in FOS, which I believe is progress! (At least it knows there is an nvme disk installed, even if FOS can’t access it).

Here are the logs from version D.

bzImage41713D_lsmod.log
bzImage41713D.log

I’ll upload version E results shortly.

@george1421 I must have missed version D, my bad. I will pull it and post the results ASAP.

I will keep an eye out for E as well.

@george1421 Sorry for the delay, here is the output of lsmod on Fedora 27 Live (4.13.9-300). Let me know if anything else would be of use.

Module                  Size  Used by
fuse                  102400  3
nf_conntrack_netbios_ns    16384  1
nf_conntrack_broadcast    16384  1 nf_conntrack_netbios_ns
xt_CT                  16384  1
ip6t_rpfilter          16384  1
ip6t_REJECT            16384  2
nf_reject_ipv6         16384  1 ip6t_REJECT
xt_conntrack           16384  21
ip_set                 36864  0
nfnetlink              16384  1 ip_set
ebtable_nat            16384  1
ebtable_broute         16384  1
bridge                143360  1 ebtable_broute
ip6table_nat           16384  1
nf_conntrack_ipv6      20480  12
nf_defrag_ipv6         36864  1 nf_conntrack_ipv6
nf_nat_ipv6            16384  1 ip6table_nat
ip6table_mangle        16384  1
ip6table_raw           16384  1
ip6table_security      16384  1
iptable_nat            16384  1
nf_conntrack_ipv4      16384  12
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_nat_ipv4            16384  1 iptable_nat
nf_nat                 28672  2 nf_nat_ipv6,nf_nat_ipv4
nf_conntrack          131072  9 nf_conntrack_ipv6,nf_conntrack_ipv4,nf_conntrack_broadcast,nf_conntrack_netbios_ns,xt_CT,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat
libcrc32c              16384  2 nf_conntrack,nf_nat
iptable_mangle         16384  1
iptable_raw            16384  1
iptable_security       16384  1
ebtable_filter         16384  1
ebtables               32768  3 ebtable_filter,ebtable_nat,ebtable_broute
ip6table_filter        16384  1
ip6_tables             28672  5 ip6table_mangle,ip6table_filter,ip6table_security,ip6table_raw,ip6table_nat
snd_hda_codec_hdmi     49152  1
intel_rapl             20480  0
x86_pkg_temp_thermal    16384  0
intel_powerclamp       16384  0
nouveau              1638400  12
coretemp               16384  0
snd_hda_codec_realtek    94208  1
kvm_intel             200704  0
snd_hda_codec_generic    73728  1 snd_hda_codec_realtek
kvm                   585728  1 kvm_intel
mxm_wmi                16384  1 nouveau
snd_hda_intel          40960  6
i2c_algo_bit           16384  1 nouveau
irqbypass              16384  1 kvm
snd_hda_codec         126976  4 snd_hda_intel,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec_realtek
ttm                    94208  1 nouveau
drm_kms_helper        159744  1 nouveau
snd_hda_core           81920  5 snd_hda_intel,snd_hda_codec,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec_realtek
dell_wmi               16384  0
drm                   352256  15 nouveau,ttm,drm_kms_helper
intel_uncore          122880  0
dell_smbios            16384  1 dell_wmi
snd_hwdep              20480  1 snd_hda_codec
sparse_keymap          16384  1 dell_wmi
snd_seq                65536  0
intel_rapl_perf        16384  0
video                  40960  2 dell_wmi,nouveau
iTCO_wdt               16384  0
dcdbas                 16384  1 dell_smbios
snd_seq_device         16384  1 snd_seq
snd_pcm                98304  4 snd_hda_intel,snd_hda_codec,snd_hda_core,snd_hda_codec_hdmi
mei_wdt                16384  0
wmi_bmof               16384  0
iTCO_vendor_support    16384  1 iTCO_wdt
tpm_tis                16384  0
dell_smm_hwmon         16384  0
tpm_tis_core           20480  1 tpm_tis
tpm                    53248  2 tpm_tis,tpm_tis_core
snd_timer              32768  2 snd_seq,snd_pcm
wmi                    24576  4 dell_wmi,wmi_bmof,mxm_wmi,nouveau
snd                    81920  22 snd_hda_intel,snd_hwdep,snd_seq,snd_hda_codec,snd_timer,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_seq_device,snd_hda_codec_realtek,snd_pcm
intel_lpss_acpi        16384  0
intel_lpss             16384  1 intel_lpss_acpi
ioatdma                53248  0
soundcore              16384  1 snd
mei_me                 40960  1
shpchp                 36864  0
i2c_i801               24576  0
mei                   102400  3 mei_me,mei_wdt
dca                    16384  1 ioatdma
vfat                   20480  1
fat                    65536  1 vfat
squashfs               53248  1
hid_apple              16384  0
8021q                  32768  0
garp                   16384  1 8021q
mrp                    20480  1 8021q
stp                    16384  2 garp,bridge
llc                    16384  3 garp,bridge,stp
crct10dif_pclmul       16384  0
crc32_pclmul           16384  0
nvme                   32768  0
crc32c_intel           24576  1
e1000e                245760  0
ghash_clmulni_intel    16384  0
nvme_core              45056  1 nvme
serio_raw              16384  0
ptp                    20480  1 e1000e
pps_core               20480  1 ptp
uas                    24576  0
usb_storage            69632  2 uas
sunrpc                331776  1
scsi_transport_iscsi    94208  0
loop                   28672  6
Module                  Size  Used by
fuse                  102400  3
nf_conntrack_netbios_ns    16384  1
nf_conntrack_broadcast    16384  1 nf_conntrack_netbios_ns
xt_CT                  16384  1
ip6t_rpfilter          16384  1
ip6t_REJECT            16384  2
nf_reject_ipv6         16384  1 ip6t_REJECT
xt_conntrack           16384  21
ip_set                 36864  0
nfnetlink              16384  1 ip_set
ebtable_nat            16384  1
ebtable_broute         16384  1
bridge                143360  1 ebtable_broute
ip6table_nat           16384  1
nf_conntrack_ipv6      20480  12
nf_defrag_ipv6         36864  1 nf_conntrack_ipv6
nf_nat_ipv6            16384  1 ip6table_nat
ip6table_mangle        16384  1
ip6table_raw           16384  1
ip6table_security      16384  1
iptable_nat            16384  1
nf_conntrack_ipv4      16384  12
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_nat_ipv4            16384  1 iptable_nat
nf_nat                 28672  2 nf_nat_ipv6,nf_nat_ipv4
nf_conntrack          131072  9 nf_conntrack_ipv6,nf_conntrack_ipv4,nf_conntrack_broadcast,nf_conntrack_netbios_ns,xt_CT,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat
libcrc32c              16384  2 nf_conntrack,nf_nat
iptable_mangle         16384  1
iptable_raw            16384  1
iptable_security       16384  1
ebtable_filter         16384  1
ebtables               32768  3 ebtable_filter,ebtable_nat,ebtable_broute
ip6table_filter        16384  1
ip6_tables             28672  5 ip6table_mangle,ip6table_filter,ip6table_security,ip6table_raw,ip6table_nat
snd_hda_codec_hdmi     49152  1
intel_rapl             20480  0
x86_pkg_temp_thermal    16384  0
intel_powerclamp       16384  0
nouveau              1638400  12
coretemp               16384  0
snd_hda_codec_realtek    94208  1
kvm_intel             200704  0
snd_hda_codec_generic    73728  1 snd_hda_codec_realtek
kvm                   585728  1 kvm_intel
mxm_wmi                16384  1 nouveau
snd_hda_intel          40960  6
i2c_algo_bit           16384  1 nouveau
irqbypass              16384  1 kvm
snd_hda_codec         126976  4 snd_hda_intel,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec_realtek
ttm                    94208  1 nouveau
drm_kms_helper        159744  1 nouveau
snd_hda_core           81920  5 snd_hda_intel,snd_hda_codec,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec_realtek
dell_wmi               16384  0
drm                   352256  15 nouveau,ttm,drm_kms_helper
intel_uncore          122880  0
dell_smbios            16384  1 dell_wmi
snd_hwdep              20480  1 snd_hda_codec
sparse_keymap          16384  1 dell_wmi
snd_seq                65536  0
intel_rapl_perf        16384  0
video                  40960  2 dell_wmi,nouveau
iTCO_wdt               16384  0
dcdbas                 16384  1 dell_smbios
snd_seq_device         16384  1 snd_seq
snd_pcm                98304  4 snd_hda_intel,snd_hda_codec,snd_hda_core,snd_hda_codec_hdmi
mei_wdt                16384  0
wmi_bmof               16384  0
iTCO_vendor_support    16384  1 iTCO_wdt
tpm_tis                16384  0
dell_smm_hwmon         16384  0
tpm_tis_core           20480  1 tpm_tis
tpm                    53248  2 tpm_tis,tpm_tis_core
snd_timer              32768  2 snd_seq,snd_pcm
wmi                    24576  4 dell_wmi,wmi_bmof,mxm_wmi,nouveau
snd                    81920  22 snd_hda_intel,snd_hwdep,snd_seq,snd_hda_codec,snd_timer,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_seq_device,snd_hda_codec_realtek,snd_pcm
intel_lpss_acpi        16384  0
intel_lpss             16384  1 intel_lpss_acpi
ioatdma                53248  0
soundcore              16384  1 snd
mei_me                 40960  1
shpchp                 36864  0
i2c_i801               24576  0
mei                   102400  3 mei_me,mei_wdt
dca                    16384  1 ioatdma
vfat                   20480  1
fat                    65536  1 vfat
squashfs               53248  1
hid_apple              16384  0
8021q                  32768  0
garp                   16384  1 8021q
mrp                    20480  1 8021q
stp                    16384  2 garp,bridge
llc                    16384  3 garp,bridge,stp
crct10dif_pclmul       16384  0
crc32_pclmul           16384  0
nvme                   32768  0
crc32c_intel           24576  1
e1000e                245760  0
ghash_clmulni_intel    16384  0
nvme_core              45056  1 nvme
serio_raw              16384  0
ptp                    20480  1 e1000e
pps_core               20480  1 ptp
uas                    24576  0
usb_storage            69632  2 uas
sunrpc                331776  1
scsi_transport_iscsi    94208  0
loop                   28672  6

@george1421 Ok, I scripted data collection and pulled fresh info for kernels a, b, c, and F27 Live. I also pulled dmidecode, full lshw ouput, and /proc/diskstats to additional log files. Let me know if they may be of any use I will upload them as well.

bzImage41713a.log
bzImage41713b.log
bzImage41713c.log
F27_4139300.log

@george1421 I had noticed that as well, and double checked the Raid settings. It had been turned back on at some point (probably while I was going through everything else). I turned it back off and re-did the debug task with the same results.

It appears that the NVMe PCIe drives use an entirely separate controller from SATA drives. Apparently the specific controller/driver varies based on the interface between the MoBo and SSD. FOS can see the M.2 drives when the are connected via a standard PCIe adapter card, however when the same M.2 drive is connected via the FlexBay MiniSAS backplane, FOS no longer recognizes the drive. This holds true regardless of the SATA controller configuration–the drives are visible with both Raid & AHCI modes.

I found a dell support article that gives some good information about the differences between NVMe and AHCI here.

I checked through the FOS kernel, and found NVME Support is enabled in lines 880-890: https://github.com/FOGProject/fos/blob/master/configs/kernelx64.config#L880

#
# NVME Support
#
CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_FABRICS=y
CONFIG_NVME_FC=y
CONFIG_NVME_TARGET=y
CONFIG_NVME_TARGET_LOOP=y
CONFIG_NVME_TARGET_FC=y
CONFIG_NVME_TARGET_FCLOOP=y

Which explains why the drives are detected when plugged into a PCIe adapter. The MiniSAS connection is a completely different beast–or at least uses a different controller from the standard NVMe drive.

I will run through the debug again (with Raid off in bios) and post the results of lspci -nn, lsblk, uname -a, and anything else you suggest. I will also do the same thing with the M.2 plugged in through a PCIe adapter so we have comparison data.

Good Morning @george1421. I stopped by the office to test the custom kernel (I’m going to pick up breakfast for my wife on my way home so she should be happy ;). No luck. The first attempt to boot gave init error, and I realized it was trying init_32.xz instead of init.xz. After adding init.xz to the "Host Init " parameter in the host record it booted into debug without issue. Bad news is that

b3:00.0 Non-Volatile memory controller [0108]: Device [1c5c:1527]

does not show up with lspci -nn and only /dev/sdb (pxe boot usb) shows up with lsblk.

@george1421 Agreed. It looks like the replacements will not ship until 8/14 so we have a bit of time for testing.

What baffles me is that I have been in contact with multiple Dell Reps trying to determine exactly which hardware needs to be returned, and none have been able to answer what part is what or point me to any product information for this drive type. As far as I can tell from their site, this particular drive configuration is not currently offered as a retail addition. The specs related to the drive based on the dell support Service Tag lookup lists only:

414-BBBV : PCIe SSD (Front PCIe FlexBay)
Part Number Quantity Description
5G90D 1 INFORMATION, FRONT, PERIPHERAL COMPONENT INTERCONNECT EXPRESS , SOFTWARE SUPPORT DISKETTE, BOOT

401-ABJT : M.2 512GB PCIe NVMe Class 40 S olid State Drive
Part Number Quantity Description
XMW6J 1 SSDR, 512G, P34, 80S3, HYNIX, PC401

400-AVDR : Dell M.2 carrier
Part Number Quantity Description
66XHV 1 ASSEMBLY, DRIVE, BAY (DRIVE BAY), M.2, MODULE

I have looked high and low for any information on these part numbers with no luck.