UEFI PXE on Dell Optiplex 7010 hangs

[JGallo]

@bardwood

Sorry I misread your post. I meant to say is we use 7010’s and I am currently on the working branch. I use legacy mode instead of UEFI and it works fine. Are you trying to stay with UEFI?

[BardWood]

@jgallo Yea. It works fine with legacy and undionly.kpxe but some of the features in the Win10 Fall Creators Update only work with UEFI. I don’t think this is a strictly FOG-related issue but some of the guys in this thread https://forums.fogproject.org/topic/4728/dell-7010-lenovo-l530-with-uefi-enabled-won-t-network-boot/59 went nuts and started building their own binaries for the 7010. Doesn’t look like the issue was ever resolved though or even identified. That thread is from 2015 so I was hoping someone had gotten it to work since UEFI boot to disk works fine on the 7010.

[JGallo]

@bardwood

I ran into an Windows 10 Creators Update issue with FOG that had to do with the bitlocker message on FOG. The FOG team pushed some updates and it works now on the working branch. Then again I only was on legacy mode and I was working with a 9020 also. Can’t say if the issue is resolved with UEFI now.

[george1421]

@bardwood Can you test something. Since you are in uefi mode, go into the firmware and change the disk mode from raid-on to ahci. I don’t think that is your issue, just yet. Using iPXE kernels not delivered by the FOG Devs is not advised. There are ipxe scripts embedded into the ipxe.efi (as well as all other fog delivered ipxe kernels) that are needed to boot into fog.

I’ve got some 7010s in my lab that I can test on Monday. We only use legacy mode at the moment so I don’t have first hand experience with uefi mode on these systems. I can say the 7010s are the second generation that supported uefi mode (in the 7xx series) and the first generation that supported pxe booting in UEFI mode, if I remember correctly. FWIW, The 790s did support uefi mode, but no pxe booting. We had to create a usb drive with iPXE installed to jump start into fog.

Anyway the issue here is that iPXE is trying to configure the hardware and its getting hung. This is typically related to faulty firmware. And frankly there is little motivation for Dell to resolve this issue since they’ve moved on to selling new hardware.

[Sebastian Roth]

@BardWood We have seen hangs on iPXE on a couple of different hardware. Your’s sounds like a buggy firmware from what I have heard so far. The actual reason we cannot tell til we dig into this. I am more than happy to go through the whole debugging process with you but only if you are keen enough to go through the whole thing with me? Can take two weeks or four weeks or just a few days of sending messages forth and back (try this, yeah shows this on the screen, picture, then try this, yeah… and so on). There is no point in starting this time consuming work if it stops right in the middle of nowhere because you had to give all the devices to users or have no time to test and send pictures anymore because your boss tells you to work on something else. So let me know.

Read through this to get an idea of the endeavor: https://forums.fogproject.org/topic/6525/pxe-boot-hp-x2-210-hybrid-tablet-windows-10-pro

As a start try using other binaries like snp.efi or snponly.efi. You could be lucky with those. We never know before we test.

[BardWood]

Thank you Sebastian, yes I can. I’ve been doing all the images/pxe at my work with FOG & CloneZilla for ~3 years and ghosting in general for ~20.

I’ll outline an infrastructure high level doc so you know what we’re working with. I technically have untill March (no hard deadline) and it’s a ‘priority project’ so this IS what I’m supposed to be doing. The infrastructure is very stable. I am a Sr admin but don’t have direct access to the Cisco network gear. I do ‘own’ the servers though. The net admin is very responsive. That said, I do general IT for a small yet global company and stuff happens.

Although I said all gens of X1 Carbon are working fine with UEFI, the original X1 isn’t working with ipxe.efi but I didn’t test other *.efi. UEFI does work but not from PXE. It’s a great loaner laptop due to built-in ETH port which I’d like to learn how to diagnose, in addition to the Dell. Seems like a great skill to have. So what do I need? Wireshark? Other tools?

[Sebastian Roth]

@BardWood Great to hear! So let’s get right into it. Please let us focus on one hardware at a time. This thread’s topic is Dell Optiplex 7010 and so we head for this. Will get to the X1 when we are done with the Dell. Just don’t want to mix up things as I am pretty sure the issues are different.

So please try PXE booting with snp.efi or snponly.efi (instead of ipxe.efi). Does this help? Just wanna make sure before we head into the ugly debugging part of this story.

@BardWood said:

So what do I need? Wireshark? Other tools?

Awesome! Seems like you know what you are doing! This will be great fun to work on I am sure. For now we need iPXE and the compiler, no network analyzing tools so far. We might get into this as well but could be that we even don’t need to. Let’s see. Are you good with C code? In case you are happy to compile and debug it yourself I can give instructions on where to look and what to do. If not we’ll do it as usually: I compile iPXE binary (debug enabled and added print out statements) and tell you where to download, you put that on your TFTP server (e.g. /tftpboot on your FOG server) and test. Take a picture from the output and screen and post it back. Then I will compile another one for the next steps and so on.

So as a start try the other binaries I suggested and if it doesn’t work change back to ipxe.efi, boot the 7010 and take a picture of the hang on screen. I am fairly sure I know how it looks like but wanna make sure there is nothing overlooked.

[Sebastian Roth]

Working on this in a chat session with @BardWood it looks as if none of the current iPXE binaries work on those machines. So we will get into debugging.

Please try 01_ipxe.efi (download - compiles with DEBUG=efi_image,image_cmd,image) and post picture or video here in the forum. Best if you use the same counter which I have in the iPXE filename for the pics/vids as well so it’s easy to reference.

[george1421]

@sebastian-roth I just grabbed a 7010 from the build up room. Let me get setup to repeat the test here.

[george1421]

@george1421 No joy with 01_ipxe.efi or stock ipxe.efi. Stops at init devices.

Its been at this screen for 20 minutes now.

(Yes I need to clean my monitor)

Update: After about 25-30 minutes iPXE appears to abort to no bootable device (Dell firmware prompt)

[Sebastian Roth]

@george1421 Nice, thanks! So here is 02_ipxe.efi… same download link.

[george1421]

@sebastian-roth No joy at the moment. My phone hates me not sending email with picture. Last call was to 0x0xd4463b88. Posted after init devices…

Here is the pict from #2

[Sebastian Roth]

@george1421 @BardWood Next to test is 03_ipxe.efi. Please post picture of the screen when it hangs…

Could be another EFI timer issue (something we had a lot of trouble with the MS Surface devices…)

[george1421]

@sebastian-roth I’m home right now, but I did update the picture. I’ll hit 03 in my AM tomorrow.

[BardWood]

@sebastian-roth
Here is from 03_ipxe.efi (after chmod 655)

Sits there for 15 mins or so and reboots

[Sebastian Roth]

@BardWood Ok, thanks! Sure I was wrong about the EFI timer, thank god.

@george1421 HAHAHAHA… I’ve been there… years ago. But couldn’t remember. LOL See here: http://forum.ipxe.org/showthread.php?tid=7877

[Sebastian Roth]

@BardWood Here is 04_ipxe.efi (compiled with DEBUG=init,runtime,efi_timer,device,pci option). Please try again and post a picture.

[BardWood]

@sebastian-roth 04_ipxe.efi:

[Sebastian Roth]

@BardWood Ok, here is ready to go 05_ipxe.efi (DEBUG=init,efi_utils,efi_driver,efi_pci). This time I guess you need to take a video of the messages passing by on screen. Make sure to rest the camera/smartphone e.g. on a pile of books so we get a decent steady picture. Would be great if you could even take a video with 60 fps which most smartphones are capable of these days!

But if there’s too much output on screen and you can’t take a proper video you might test 06_ipxe.efi (DEBUG=init,efi_utils,efi_pci)…

@george1421 Re-reading all the stuff we tried back in 2015 I get the impression that the issue with those 7010s is that for some reason the native Intel driver is not being used. Then it tries SNP and for some reason that fails. More or less a guess though.

[BardWood]

@sebastian-roth

RE: 05 = Wowsa! Yea, much output. My phone won’t let me video yet because the battery is too low (???). I’ll capture a video later but I cant even read it in person it scrolls by really fast. I’ll do what I can.

RE: 06 =