No network interfaces found | Dell PowerEdge R650 | Intel E810-XXV

[djgalloway]

I just got a new batch of Dell R650 servers with both an Embedded BCM5720 NIC and an add-on Intel Ethernet 25G 2P E810-XXV. I will not be cabling the 1Gb NIC and intend to only use the 25Gb NIC.

The servers are able to PXE boot to the 25Gb NIC and even get the FOG menu but upon attempting to capture a FOG image, the kernel quits at “No network interfaces found.” This is after I disabled the 1Gb NIC in the BIOS.

This tells me the kernel doesn’t have the right driver but I am using the latest kernel available.

file /var/www/html/fog/service/ipxe/bzImage*
/var/www/html/fog/service/ipxe/bzImage:   Linux kernel x86 boot executable bzImage, version 5.15.34 (buildkite-agent@Tollana) #1 SMP Sat Apr 16 10:44:28 CDT 2022, RO-rootFS, swap_dev 0x8, Normal VGA
/var/www/html/fog/service/ipxe/bzImage32: Linux kernel x86 boot executable bzImage, version 5.15.34 (buildkite-agent@Tollana) #1 SMP Sat Apr 16 10:40:09 CDT 2022, RO-rootFS, swap_dev 0x8, Normal VGA

As I understand it, they should be supported https://github.com/torvalds/linux/commit/7dcf78b870be6418d72bb1c4d4924bf0f5ca5052

[george1421]

@djgalloway we are going to need to build a one off kernel for this. We can get an answer but will need your help. But to preface this FOG is intended and designed to image desktop or workstation class systems. Its not designed for server with custom and unique hardware requirements. With that said fog can be extended to support almost any intel based computer.

I think I might have build a custom kernel for a server build back in june. I see this kernel in my inventory with qed in the title. At the time it meant something to be but now… I think this was the custom kernel I built for an other fog admin. A quick google search returns this linux kernel network adapter “QLogic QED 25/40/100Gb core driver”

https://drive.google.com/file/d/1p6sJyIv8uDhF4IUAo7uMzoSp9BmkLzQV/view?usp=sharing

So how do you use this?

Download this file as bzImage5.15.44qed-2 Watch your case, because it is important. Copy this file to the fog server in /var/www/html/fog/service/ipxe directory.

Now go into the FOG WebUI ->FOG Configuration -> FOG Settings. Hit the expand all button, then search the page for bzImage. There will be a bzImage and bzImage32. We are interested in bzImage. This is the global setting that tells FOG what kernel you want to load. We will change the global because these R650 computers haven’t been registered yet. In the field that says kernel and has bzImage as the value in the property field, change that to bzImage5.15.44qed-2 the same exact name and case you set the file to. Now save the settings.

Now pxe boot into the FOG menu. Pick full registration, or hardware detection. Have a quick eye, but you should see ipxe download bzImage5.15.44qed-2. If it does you know you have the updated kernel. See if that picks up an IP address.

If that doesn’t work then we will debug this server to get the answer. I’m only offering this kernel because I think this has the right drivers loaded into it.

edit: found it, in this thread it was a R640 server https://forums.fogproject.org/topic/16244/fog-cannot-load-qlogic-network-driver-on-poweredge-r640 Now that I look at the subject line, it looks like you have an intel network adapter and not a qlogic adapter, so this probably won’t work. We will have to go the debugging route.

[djgalloway]

@george1421 said in No network interfaces found | Dell PowerEdge R650 | Intel E810-XXV:

@djgalloway we are going to need to build a one off kernel for this. We can get an answer but will need your help. But to preface this FOG is intended and designed to image desktop or workstation class systems. Its not designed for server with custom and unique hardware requirements.

I’m aware of this and grateful for any special assistance you can provide. FOG is the perfect tool for my use case and has been serving us faithfully for 5 years.

We will have to go the debugging route.

Sure. How can I help?

[george1421]

@djgalloway OK for debugging its pretty easy on your part (kind of).

You are going to connect that server to the network for pxe booting via the sfp+ port. You need the mac address of that network port you are using.

In fog manually register that host (in host management) with the collected mac interface. Now schedule a deploy image to that computer, we are not going to deploy anything during the debugging process. Before you hit the schedule task button tick the debug check box then schedule the task.

PXE boot the target computer, it should go right into imaging where you will not see the fog ipxe menu. After a few screens of text that you will need to clear with the enter key you will be dropped at the fos linux command prompt on the target computer.

Now I want you to run these debugging commands to help identify that hardware. Take a screen shot of the output. I’m most interesting in the hex codes that appear between the squared brackets on the next command.
lspci -nn | grep -i net

This should list all of the network adapters installed in that computer. We should see the LOM adapters and then the OPC or pcie adapters. I need the hex code to know what kernel model is missing.

The next command we need to search the syslog to see if there are any messages regarding missing firmware.

grep -i firm /var/log/syslog (check to confirm syslog file exists in /var/log directory. That should return any lines where firm is mentioned. For certain network adapters updated firmware (microcode) may be required. These need to be included in the custom kernel.

The next commands have nothing to do with the network but rather with the disk subsystem. Lets see if FOS Linux can see the disks in this server. (your next battle once network works).

lsblk
lspci -nn | grep -i raid

[djgalloway]

@george1421 Here you are!

Looks like it’s playing nice with the RAID controller at least.

[george1421]

Don’t worry about understanding this next part, these are notes on the hardware I need.

[8086:159b] “Ethernet Controller E810-XXV for SFP” first appeared in linux kernel 5.7.x, not currently included in FOS Linux kernel. One off kernel required.

Kernel config parameter: CONFIG_ICE

Firmware says ice director firmware for (something) for intex/ice/ddp/ice.pkg failed with error -2 [hint needs firmware]

raid controller is LSI SAS39xx [1000:10e2] lsblk sees 7 disks in non-raid format or software raid.

**It looks promising to build a custom kernel for this hardware.

[george1421]

@george1421 Here is the link to the one-off kernel :https://drive.google.com/file/d/18pWcju562KfM-iFbkD86ylWM5R2YxgK5/view?usp=sharing

This one is called bzImage5.15.44-ice Looking at the custom kernel I had the ice driver installed but was missing the firmware bit. So this kernel has CONFIG_ICE enabled as well as preloading the firmware intel/ice/ddp/ice-1.3.26.0.pkg into the kernel.

For installing it use the same instructions as the qed-2 kernel below, use the -ice name instead.

A short cut here is in the host definition you just created, there is a kernel field. Update that field to bzImage5.15.44-ice, then save the host profile. Reboot the target computer (assuming its still in debug mode). Have a quick eye, as you should see it now transfer bzImage5.15.44-ice to the target computer as it pxe boots. At the FOS Linux command prompt key in ip a s to see if it gives you an IP address now. If no report back and we will continue debugging.

[djgalloway]

Still no dice unfortunately. The NIC doesn’t show up in ip output at all.

[Sebastian Roth]

@george1421 Looks like the DDP binary blob needs to be named as ice.pkg without the version number (code ref). I don’t think I have seen this ever before and I am not sure the buildroot init is able to properly handle this.

Could you try manually renaming the file after cloning the firmware repo and adjust the filename within the kernel config?

[george1421]

@djgalloway OK updated version: https://drive.google.com/file/d/18jEOG-qqTpJOS6NzfBRzDXBu1E7O9ZHt/view?usp=sharing

Save as bzImage5.15.44-ice2

I did exactly what Sebastian mentioned. I copied the .pkg file to ice.pkg and did a complete rebuild on the kernel thats why it took a bit longer to complete.

[george1421]

@sebastian-roth said in No network interfaces found | Dell PowerEdge R650 | Intel E810-XXV:

I don’t think I have seen this ever before and I am not sure the buildroot init is able to properly handle this.

Same here on the naming convention. I’m building it outside of build root, but in a kernel dev environment so I have a bit more flexibility to create mischief.

[djgalloway]

@george1421 That did it!

Is there any additional information I could provide that might be helpful?

Is there any harm in replacing my existing bzImage files with this patched kernel?

[george1421]

@djgalloway said in No network interfaces found | Dell PowerEdge R650 | Intel E810-XXV:

Is there any additional information I could provide that might be helpful?

No that should do it if it sees the nic and it gets an IP address. Just remember to remove your debug deploy task and you should be done. It sees the disks so it should image now

Is there any harm in replacing my existing bzImage files with this patched kernel?

No harm in replacing your current default image. I would go in and change the global kernel parameter in the fog configuration -> fog settings -> expand add method. You just have to remember you are running under a one off kernel when/if you do a kernel update since the default kernel is something other than bzImage. But this kernel is the same that as what is released by the developers with the addition of the qed and ice spf+ drivers.

[djgalloway]

@george1421 Great, thanks much for your help. I don’t have a lot of spare cycles to donate to the project at the moment but I did just throw you all some beer money.

[george1421]

@djgalloway Thank you. Actually this thread helps advance the project too. The devs won’t add this driver to the main kernel but if they need to create a one off kernel the instructions are here.