iPXE won't find network cards

[mckayj]

We have purchased a SuperMicro 4029GP-TRT to be used for scientific computing.
We are using FOG to distribute our OS image, and have had no problems with other computers (except issues that were previously resolved on this forum).
However, this 4029 refuses to boot the iPXE. Here is what happens:
-Computer starts, and starts to boot PXE.
-PXE gets DHCP address, then switches to iPXE.
-iPXE does not get an address, fails, and reboots after 10 seconds.
It seems the iPXE simply doesn’t recognize the network card.

More details:
When iPXE fails, I can select “s” to enter a minimal shell. If I enter the command “ifstat”, it does not show the Ethernet network card.
-(It shows our Infiniband network card, but we’re not booting over that.)
I have tried changing the ipxe image selected by the DHCP server to all the other files provided for PXE. None of them work.
The network card is an on-board adapter, from the Intel C622 chipset.

This is a pretty new system, and a 10gig network card.
Has anyone seen this before?

Thank you

[george1421]

Is this system booting uefi or bios mode?

What boot image are you using (dhcp option 67 if using static boot files)?

You have to be using the right ipxe boot file based on the mode or iPXE would simply fail to start.

[mckayj]

Thanks for the speedy response.

I believe the network cards are booting the old BIOS mode (They are set to “Legacy” in system setup. If I change them to “EFI”, it works even less well).

The boot image is set to ipxe.pxe, although I’ve tried several different files. None work better.

I’m pretty sure the iPXE is starting:
PXE starts and gets and address, then the normal startup messages for iPXE print.
But when it normally (on other computers) says that it is configuring net1, it just says DHCP fails.

Screen shot:

Looking at the screen shot, you can see the standard network boot, and DHCP, and then it switching to iPXE.
iPXE then tries net0, however this is our Infiniband network, not our Ethernet network, so it can be ignored.
It then fails.
I can then run the ifstat utility, and it only shows the Infiniband, no other network cards.

All our other nodes have both Infiniband as well as Ethernet, and they work fine; the iPXE uses “net1”.

Hope this helps.

[Sebastian Roth]

@mckayj Could be a spanning tree issue. Can you put a dumb mini switch between client and switch? I see 10gig mentioned. You probably can’t then. Do you have a access to configure the switch? If so make sure to set this particular port to “fast port” (aka non-spanning tree, other vendors might call it different).

[george1421]

@mckayj Since its not seeing the network adapter in iPXE, I think that is the key.

So if you are in bios mode you may be sending undionly.kpxe to the target computer via dhcp option 67.

This driver relies on the network card’s built in undi driver. I’m wondering what would happen if you changed the iPXE boot kernel from undionly.kpxe to ipxe.kpxe then tried to pxe boot from there. The ipxe.kpxe includes all of the known iPXE network drivers vs undionly only including the undi network interface. Its complicated to explain other than just change your ipxe boot loader to ipxe.kpxe and see if you have better success.

[mckayj]

@george1421 I have tried ipxe.pxe, ipxe.kpxe, and ipxe.kkpxe, as well as some of the Intel, Realtek, and UNDIonly variants.
None work, all refusing to see the Ethernet network card
(And, the Intel and Realtek ones won’t see the Infiniband card either, but that doesn’t really matter)

[george1421]

@mckayj Well that is unfortunate none of the other iPXE kernels support that network adapter. If iPXE doesn’t have a built in driver, I’m suspecting that FOS may not see that network adapter either. From a Windows system can you get the hardware id (vendor ID and model) for that network adapter. You can do that from the device manager, inspecting the network card properties and then selecting hardware ID.

So at this point I think you have 2 options.

1. Get a USB3 to ethernet adapter that supports pxe booting.
2. Attempt to usb flash drive boot right into FOS bypassing iPXE altogether.

Both options have their caveats, but (as long as FOS supports the NIC) you should be able to boot and image with either.

[mckayj]

@Sebastian-Roth They are managed switches, but pretty much all settings are in their defaults - and, I’m not really familiar with with Spanning Trees, or how to set them up with these switches.
I can however confirm that the network cards are linking up, and that the first stage DHCP for PXE gets an address.

[mckayj]

@george1421 I managed to find a PCIe network card in an old server on its way to the junk heap. Salvaged that, and placed it in the node in question. iPXE worked fine with it.
So, I guess the moral of the story is some super new network cards iPXE can’t talk to.

[george1421]

@mckayj said in iPXE won't find network cards:

some super new network cards iPXE can’t talk to

I’m more suspicious of 10G nics than super new cards. I don’t know off the top of my head if iPXE supports 10G cards. Or it may be a combination of super new and 10G. ??

As long as you have a path forward then use it.