Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized

[hlalex]

@george1421 I have to get these drives shipped out by 3:30pm EST today. I anyone has any additional test let me know and I will get anything I can.

Otherwise, Thanks for all the help!

[george1421]

@hlalex Thank you for your help. Go ahead and send them back at this time. It looks like kernel changes between 4.13.x and 4.17.x has disabled these types of drives. My only option would be to see if 4.18 has fixed the issue. But at this point you have a solution and the hardware must go back. So we’ll park testing until someone else has this issue. Again thank you for all of your testing and data collection. We would not have made it this far without your help.

[bright23]

I have this same issue and have several machines in stock to test with if needed. However, I’m not nearly as Linux savvy as hlalex. I appreciate the effort you two have already put into it.

[george1421]

@bright23

There are a few more ideas I’ve had.

First make sure your system is configured for uefi and have secure boot disabled, as well as have legacy roms disabled in the firmware.

Well if you are willing to try the 4.18.3 kernel I built yesterday, we can continue testing. It does seem to be that something broke (changed) after linux kernel 4.13.x

Here is the 4.18.3 kernel I built yesterday: https://drive.google.com/open?id=1VVXEVY0p2Gl2w4WvT_0jstOtV1jLBp1k

1. Download that file to /var/www/html/fog/service/ipxe directory on the fog server.
2. Register one of these ‘troubled’ workstations
3. Go into the host management of this ‘troubled’ workstation and in the Host Kernel field enter bzImage4183
4. Schedule an image deployment, but before you submit the deployment task, tick the debug check box.
5. PXE boot the target computer, it should go right into deployment.
6. After several enter key presses on the target computer you should be dropped to a linux command prompt.
7. At this point I just want you to key in one command lsblk. I need to know if you see something that is similar to /dev/nvme0n1 listed as one of the block devices.

Try the above steps with

I’m in the process of rebuilding the inits to support Linux kernel 4.13.x. I’ll have another test in about 1 hr to try and older kernel with the current FOS inits. This is a lame attempt to get things moving until we can get a better understanding of what we are seeing here.

[george1421]

Something I want you try first before the new kernel, read through my previous post to understand where/what I’m talking about.

For this first test, I want you to go into host management for one of these troubled systems and in the Kernel Parameters field add nvme_load=YES Using the stock out of box FOS kernels.

Setup a debug deployment as below and see if lsblk returns the nvme drive.

[bright23]

@george1421 sorry I just had a moment to get back to you. I’m running Kernel 4.17.0 and have “Host Kernel Arguments” = nvme_load=YES, booting into debug running lsblk returns nothing.

[bright23]

I did also confirm that it’s using uefi, that secure boot is disabled and that legacy roms are also disabled.

[bright23]

@george1421 when the trouble machine boots into the test kernel (bzImage4183) it’s unable to get past the iPXE process:

 bzImage4183... ok
 Could not select:  Exec format error (http://ipxe.org/2e008081)
 Could not boot:  Exec format error (http://ipxe.org/2e008081)
 Could not boot:  Exec format error (http://ipxe.org/2e008081)

[matijn]

Sticky, also Dell Optiplex 5820.
2x Front PCIe FlexBay, 512GB Hynix M.2 PCIe NVMe Class 40 SSD )

Any progress on the kernel?

[bright23]

I tested 4.18.3 also, no joy.

[Sebastian Roth]

@bright23 I guess @george1421 is on holiday leave at the moment as I have not heard from him in the past days. He’s been doing a brilliant job debugging this issue and I don’t think it’s wise if I get into this. Let’s hope he’s coming back soon.

[george1421]

@Sebastian-Roth I paused debugging on this issue. It appears that something changed in the linux kernel between 4.13.9 and 4.15.2 to causes these flex bays to not init correctly. Without having one of these systems in my hands its hard to debug exactly what is going on. I’m sure they moved some of the core code to an independent module.

The only thing I can think of to get by this issue immediately is to rebuild the inits and set the minimum kernel level to 4.13 to allow the 4.13.x kernels I built to use the current inits. Its not a nice answer, but until I happen to order one of these systems its hard to see what is actually going on.

[Sebastian Roth]

@george1421 said in Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized:

until I happen to order one of these systems its hard to see what is actually going on.

Either that or @bright23 you are getting into kernel compiling and debugging the issue with our help. Are you keen to?

[george1421]

@Sebastian-Roth said in Dell Precision Tower 5820 - FlexBay MiniSAS PCIe NVMe SSD not recognized:

Either that or @bright23 you are getting into kernel compiling and debugging the issue with our help. Are you keen to?

I still have my kernel build environment setup if something needs to be compiled. I can grab the latest linux kernel and rebuild it in the AM if needed.

[matijn]

@hlalex FOG seems to work when NVMe’s not plugged in ODD sata ports / Dell 8520.

N I C E

[george1421]

@matijn Thank you for providing feedback on this issue, since it has been driving us crazy not having the hardware on hand.

Just so I understand the picture here. FOG Images OK when connected to the sata ports as listed in the picture? Again for clarity the picture above shows a working solution?

I take it the ODD sata ports are the special new data ports? From within windows, can you tell what hardware ID is associated with those ODD ports? I feel that the linux kernel is missing a needed driver, but we can’t seem to identify it.

[matijn]

@george1421
What I understand, out of dell 5820 owners manual:
https://topics-cdn.dell.com/pdf/precision-5820-workstation_owners-manual2_en-us.pdf

ODD is Optical Disk Drive. Maybe wrong assembly.
I can confirm, capture and deploy task, no issue. OK.
I will update this on the hardware id question…

[Quazz]

Because they’re marked as ODD, they might be on a different controller altogether and possibly with more limited functionality too.

[hlalex]

@matijn I actually did not have issues with the ODD ports (when used with 2.5 SSDs), they connect the two SATA ports on our 5820s. The two FlexBay ports are connected to PCIE0 and PCIE1, which are a mini-sas type connector (pic below). I can’t remember if I tried moving the flexbay NVMe to the standard hot swap bays, but I don’t believe I did.

[george1421]

@hlalex This is all great info guys. Thank you.

Its hard without having the hardware in hand for the developers to find out what’s missing. Seeing is better than trying to imagine what things look like.