Problems with Optiplex 5260 in an advanced PXE environment

[pbriec]

Hi,

i’m trying to use FOG (1.5.4) with several Dell Optiplex 5260.
I can access the FOG pxe boot menu.
If i’m trying to set “Perform Full Host registration and inventory” or “Quick Registration” or anything else, i can only see
bzImage32.xz ok
init_32.xz ok
and bootloop on the FOG pxe menu.
I’ve tried several kernel, but it doesn’t change anything.
How can i solve this problem or debug it.

thanks
Pierre

[pbriec]

good job! you are the best.

it seems to work with my VM. I will try with the Opliplex 5260 very soon. I’m not on the same working place today.

thanks
Pierre

[george1421]

@george1421 The default.ipxe file on the FOG server should look like this:

#!ipxe
cpuid --ext 29 && set arch x86_64 || set arch ${buildarch}
params
param mac0 ${net0/mac}
param arch ${arch}
param platform ${platform}
param product ${product}
param manufacturer ${product}
param ipxever ${version}
param filename ${filename}
param sysuuid ${uuid}
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:bootme
chain http://<fog_server_ip>/fog/service/ipxe/boot.php##params

where <fog_server_ip> is the physical IP address of your fog server on your imaging network.

[george1421]

@pbriec Just for clarification, this file is from the FOG Server and not your main pxe boot server? We need this chain command file on your primary pxe boot server only.

You should not have touched this file, its created by the fog installer. If your post is accurate that explains why we received that error message.

[pbriec]

@george1421
only that

#!ipxe
chain tftp://172.16.4.8/default.ipxe

[george1421]

@pbriec Would you post the content of /tftpboot/default.ipxe from your FOG server? The more I look at this setup the more I think it should work as we configured it. I have to be missing something…

[george1421]

@pbriec Oh we are very close. I see it’s doing very much like we need.

1. I see that its using an updated (different) ipxe kernel since the build number has changed (960d1) and it has all of the proper modules (DNS, FTP, HTTP…) That tells me you have the FOG iPXE kernel running (goal 1).
2. Its chain loading from your main pxe boot server to the fog server (tftp://172.16.4.5 -> 172.16.4.8) So its making it to your FOG server (goal 2).

The issue is its not supplying the needed parameters along the way. I think we need to adjust the default.ipxe on your main server to detect the required parameters. We should review the embedded ipxe script from the github site to see what is missing. https://github.com/FOGProject/fogproject/blob/master/src/ipxe/src/ipxescript

The problem is very close to a solution since we are getting all of the bits in place to make this work,.

[pbriec]

Hi @george1421, thanks for taking time to solve this problem

here is my lan where x=0 for site1 and x=4 for site2. They have both the same configuration.

my DHCP/DNS server are 172.16.x.3
my FileServer/TFTPBOOT are 172.16.x.5
my FOG Server are 172.16.x.8

:call_fog
set next-server 172.16.4.8
iseq ${platform} efi && set filename ipxe.efi || set filename undionly.kpxe
chain tftp://${next-server}/${filename}

here is the output from my vm when i hit the new item call_fog

thanks
Pierre

[george1421]

@george1421 OK this problem is making my head explode.

In your main pxe boot server, I assume you have a menu entry that points to the FOG server. The issue we have is we need to replace the in memory iPXE kernel with the FOG iPXE kernels (like what you did when you changed the dhcp option 66 and 67).

So in your main pxe boot server for the menu entry for FOG see if this replaces iPXE with FOG’s iPXE.

:call_fog
set next-server <fog_server_ip>
iseq ${platform} efi && set filename ipxe.efi || set filename undionly.kpxe
chain tftp://${next-server}/${filename}

That should chain load the fog iPXE kernel. There is an embedded script in the FOG iPXE kernel here: https://github.com/FOGProject/fogproject/blob/master/src/ipxe/src/ipxescript

Its a generic script, the only thing we have to pay attention to is line 29.

chain tftp://${next-server}/default.ipxe ||

What that script will do is again acquire the pxe boot server {next-server} from dhcp, which will point back to your main pxe boot server, but ignores dhcp option 67 and calls default.ipxe. On your main dhcp server you need to create a new text file called default.ipxe and in it place this script

#!ipxe
chain tftp://<fog_server_ip>/default.ipxe

That should call the fog based ipxe boot script, you will be running under the FOG iPXE kernel done by the fist chain, and the second chain will jump you into the FOG iPXE menu.

Will it work?? Maybe /maybe not but it sounds good on paper.

[george1421]

Please understand I’m not faulting anything at the moment. What we have is a puzzle and I’m trying to find all of the bits that makeup this environment. I think there is a solution here, we just need to find the right combination.

Looking back over our chat session the OP kept saying iPXE but my intuition was telling me syslinux. Then I looked back for the dhcp config file posted.

option tftp-server-name “172.16.0.8”;
option root-path “/data/tftpboot/”;
next-server 172.16.0.8;
filename “pxelinux.0”;

    if option arch = 00:06 {
            filename "efi32/syslinux.efi";
    } else if option arch = 00:07 {
filename “efi64/syslinux.efi”;
            filename "ipxe.efi";
    } else if option arch = 00:09 {
filename “efi64/syslinux.efi”;
    } else {
            filename "pxelinux.0";
    }
}

There may be some formatting errors above that chat did to the config file, but I can clearly see syslinux kernels being called. The OP also made reference to pxelinux.cfg in the chat. Which again is a syslinux thing.

[george1421]

@Sebastian-Roth OK I have a bit more understanding what is going on here.

The OP has 2 iPXE boot servers. He has a main one that does other things already. He has setup a chain statement in his main pxe boot server to load boot.php from the FOG server. So the booting process is still under control of the iPXE kernel running from the primary pxe boot server (not-FOG server). If he updates his dhcp server to point to the FOG server this Optiplex 5260 pxe boots correctly. If he chain loads from his main pxe boot server to fog it fails. According to the OP the chain loading works with other hardware, just not with this specific model.

From his existing iPXE boot server he is using this command to chain: chain http://x.x.x.x/fog/service/ipxe/boot.php?mac=${net0/mac}

The version of iPXE running on his main pxe boot server is 1.0.0 1b104 (pretty old).

For bios based systems the OPs using pxelinux.0 for uefi based systems its ipxe.efi (on the main pxe boot server).

So to make this work, we need to get his main pxe boot server to load the proper iPXE kernels from the FOG server to support iPXE booting or possibly replace the iPXE boot kernels on the primary PXE boot server with the newer ones from the FOG server. The only question is if the built in iPXE script in the FOG iPXE kernels would cause his main pxe boot server problems.

I know I talked in a circle here. I needed to get everything down in one place do see if we can come up with a solution.

At this time I don’t feel its a FOG component level issue, but the OPs environment.

One possible solution would be to use the FOG server as the main pxe boot server and then just migrate the menus from the existing main server to the FOG server.

[george1421]

@Sebastian-Roth Just a second on chat with OP. He has a different configuration than we are expecting.

[Sebastian Roth]

@george1421 @pbriec Not sure where we are headed with this. Directly booting the kernel as UEFI binary like I suggested seemed to cause the kernel to hang just the same. So I am very confused on what exactly is being done. It doesn’t add up for me because the FOS USB boot suggested by George is very similar to what I asked you to test…

[george1421]

@pbriec I’m not sure why you’ve gone down this path. But based on your previous test FOS boots to the debug console prompt. So that tells me the issue is not with FOS, but with iPXE not being able to start FOS for some reason. We have seen this before with mainly Lenovo systems with faulty firmware (why I suggested you update the firmware).

I think we are at the point where we need to get @Sebastian-Roth involved with debugging iPXE to find out why its not launching the FOS kernel correctly.

[pbriec]

@george1421

Ok, i’ve made some changes on my systems
As i’ve already a pxe server on my enterprise, i’ve tried to swap the PXE server in the configuration so now, the FOG pxe server is the main pxe server. I’ve made changes on the dhcp server.
I’ve deleted the host with FOG gui and now, i can access the full inventory.

How can i use 2 pxe server by using pxe chainloading?
BOOT --> 1st PXE server --> chainloading to FOG iPXE server
What is the best practices?
here is what i have on the pxe menu config

:fogpxe
dhcp
cpuid --ext 29 && set arch x86_64 || set arch i386
params
param mac0 ${net0/mac}
param arch ${arch}
param platform ${platform}
param product ${product}
param manufacturer ${product}
param ipxever ${version}
param filename ${filename}
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:bootme
#chain http://172.16.6.157/fog/service/ipxe/boot.php?mac=${net0/mac}
chain http://172.16.0.8/fog/service/ipxe/boot.php?mac=${net0/mac}

in fact, all was commented except last line.
If the lines are uncommented, it does’nt boot to FOG from my iPXE server

I’m using the other pxe server for other purposes than FOG
Can you help me?

[pbriec]

@george1421

I’ve updated the bios firmware, nothing change.

I’ve tried your FOS and it is starting so the problem come from ipxe

Pierre

[george1421]

@pbriec said in Problems with Optiplex 5260:

I’ve run again the installer. It is the same

Have you confirmed that you have the latest firmware in stalled on these 5260s? That was also part of my request.

The second thing is now what you posted

Kernel.SR.SKIP_PCI.4.8.11.64... ok
init.xz... ok

Now the 64 bit kernels and inits are being sent? What changed? This is what I would have expected the first time. Did the installer fix this issue or did you specifically define init.xz when you defined the kernel to use. I’m only focusing on FOG system behavior here.

I think the next thing we need to test is if you can usb boot into FOS (FOG customized linux that runs on the target computer). FWIW FOS is the combination of bzImage and the virtual hard drive init.xz.

Review this tutorial, pay special attention to the caveats mentioned. https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image/4

Look at the FOG Forum chat bubble for some additional instructions to take you to step 7 in the tutorial. “Burn” the image onto a USB flash drive of at least 512MB in size. Edit the grub.cfg file as in the tutorial. Then boot from this USB drive and select option 6 from the GRUB menu. The whole purpose of this is to see if FOS runs on the target computer. That will tell us if the error is coming from FOS or iPXE not starting FOS correctly. I’m suspecting the issue is not with FOS, but iPXE, but that is only a guess based on what we know right now.

[pbriec]

@Sebastian-Roth

thanks for your answer
Ok, here is the output with a USB UEFI key

done
exit_boot...

[Sebastian Roth]

@pbriec Will have to check my code when I get home. Don’t have that here at the moment. But I am wondering about the third line done. Not sure where this is coming from but I can’t remember having that in my debugging output. Possibly I am wrong, will see later.

I worked with JJ on a similar issue where it seems to hang on exit_boot. But this is a HP Stream 11 x360 device. I really wonder if we see buggy UEFI firmware in both cases?!?

Can you please try booting Live Linux Distros, like Ubuntu Live, System Rescue Disk or so. Will be interesting to see if those boot properly.

The other thing you can try is booting that downloaded kernel image directly from a UEFI USB boot stick. To create one format as FAT32 and create directory X:\EFI\boot\, put the kernel binary into that and name it bootx64.efi. Now boot off this USB stick and see how far you get. If it proceeds beyond the exit_boot stage it will crash in a kernel panic later because there is no root filesystem to mount and boot off. That’s ok. We just want to see if it can get past exit_boot when started in a different way from a USB stick instead of PXE boot.

[pbriec]

@george1421

I’ve run again the installer. It is the same

@Sebastian-Roth

in fact, i’m new in the FOG world. I’ve tested the pxe with another PC and i can access the Full Inventory. With VMs, it also works. Now i have to deploy 50Pcs with Optiplex 5260 configuration :-(. I hope i will not have to deploy by hands

I don’t have to make videos as here is the output (no bootloop)

Kernel.SR.SKIP_PCI.4.8.11.64... ok
init.xz... ok
done
exit_boot..._

thanks
Pierre