@sgilbe OK so this is telling us that the linux kernel is not booting on this hardware. Can you go to this URL https://github.com/FOGProject/fos/releases and download the 5.15.x bzImage and copy that to the usb drive to see if 5.15 will boot on this hardware? If not working, then try the 5.10.x, if that fails then try the 4.19.x series.

As for the type of system it is an Intel 4th Gen Xeon system (D50DNP)
What exact processor is installed in this computer? I might have to search this issue based on the processor. You were able to live boot ubuntu 22.04 so a linux kernel should boot on this system. There might be something missing from the FOS linux kernel.

@totoro Do you have access to the FOS linux console or is the output from some other linux distro?

If you have access to the fos linux console key in the following commands and post the output here.

uname -a
lspci -nn | grep -i net
grep -i firm /var/log/syslog

1. The first command tells me the linux kernel version
2. The second command lists all network adapters in the computer. The important value missing from your printout is the device id [8086:xxxx] that will tell me if linux has a driver for it.
3. The third command should list any possible firmware the network adapter is missing that we need to include in the kernel.

@totoro Do you have the option to usb boot this device for imaging?

For completeness, you say you are able to image using 2 network cards. One allows pxe booting. What model network adapter is this? The first one is not seen by FOS linux but the second one is seen by FOS. What model number is that? Is it an added in network adapter like usb a or usb c network adapter? The transfer of 26% of bzImage is suspicious why it works for capture but fails on deploy. I have to be missing something here. I realize this hardware is not a typical PC, but if its x86 based it should work.

@sgilbe ok, please pick the debug option, I think its item #6. That menu has a log level of 7.

But it does look like the linux kernel is not booting on this hardware, hopefully the additional logging level will give us some help. Also now that you have a usb drive, we can try the 5.15.x linux kernels too. It might be something in the 6.x kernel.

@skyborne98 Will you provide a clear screen shot taking with a mobile phone of the actual error so we can see the context round the error message.

One other debugging step we can do is to schedule a capture/deploy (desn’t matter) but before you hit the schedule task button, tick the debug checkbox. Now pxe boot the target computer and it will begin imaging in debug mode. So it will drop you to the fos linux command prompt on the target computer. Key in fog to start the imaging process. The imaging process will single step through imaging. You will be required to press enter at each breakpoint to continue. When you get the error press ctrl-C and then take a screen shot of the error. From there we can begin trying to debug what is going wrong.

One thing that also reminds me, did you update the database schema? It would have been a step in the installation process to go to the web gui and update the schema.

@iljared98 The problem here is raid-on + uefi + linux. There is no known driver for raid-on mode with uefi and linux. You need raid-on mode for the intel raid controller to work for the fake raid.

For full disclosure I have not tested the intel hardware accelerated raid with the 6.x linux kernel yet.

Now if you change one of the settings you will be able to image. Something you can try. I know this works, but I have not tried it recently or with your hardware. If you have a uefi image, you can capture and deploy that image using a bios based computer, as long as the first boot is in uefi mode of the target system. So the idea here is to pxe boot the computer from the Dell boot manager in bios mode (CSM mode needs to be enabled in the bios). Deploy your image to the raid disks in bios mode and see what that gives you. Now I’m suggesting that you leave the computer in uefi mode with raid-on just from the boot manager pxe boot in bios mode (hopefully that is still an option). Upon a reboot after imaging the computer will remain in uefi mode and for the OS first boot it will be in uefi mode.

@sgilbe said in Client hangs at EFI stub::

I will go through the forum chat to see if there is anything there about the issue

Hint: little speech bubble along the fog menu bar on the right next to your avatar picture.

@skyborne98 There as been many, many changes since 1.2.x series. One notable change is that the service account fog has changed to fogproject, but that would only come into play during a capture.

The permission denied message during a deploy is suspicious since FOS linux (at this point in a deployment) is running as root. It should have access to the nfs share.

Also for reference, the .fogsettings file is only referenced during an install.

Lets have you run through this tutorial. Remember the service account for FOG is now called fogproject since there were bad instructions on the internet for people to use the fog account to install fog, which reset its passwod. https://forums.fogproject.org/topic/11203/resyncing-fog-s-service-account-password?_=1694461880173 Lets make sure your fog service account is all synchronized before we dig deeper.

@sgilbe I can say I have not tested this process with Rocky Linux (any version). I did create the image for the tutorial using ubuntu/debian version. This issue may be related to a newer version of grub. You might want to try to add the --force switch to the command.

Did you look at the forum chat at the top of the page for additional hints?

@elchapulin This should work for pxe booting except the 32 bit field is bad. It should be i386/ipxe.efi but that probably isn’t your problem since 32 bit uefi systems are rare.

So when you say the pxe client can’t boot, what errors do you see?
Is the fog server, dhcp server and pxe booting client on the same subnet? If yes then we should grab a pcap of the pxe booting process from the fog server using: https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue?_=1694459465428 You can look at the output with wireshark.

In the DISCOVER statement in dhcp options 93 or 94 the pxe booting computer will tell its arch. The dhcp server will take that statement and then (should) seen the proper boot file info in dhcp options 67 as well as the boot-file field in the ethernet header of the OFFER statement. The OFFER statement comes from the dhcp server. If you have only pfsense as your dhcp server then you should have only one OFFER statement, if you see more than one you have multiple dhcp servers on your network.

If you can’t figure it out with wireshark, upload the pcap to a file share site and post the link here or DM me the link with FOG forum chat. I’ll take a look at it.

Also one additional comment, if you have uefi system make sure secure boot is turned off or the firmware will reject ipxe from loading.

@elchapulin Yes it does. (saying these instructions from memory) in the dhcp server there is a section under advanced for netbooting (its not called pxe booting). There was 4 or 5 fields. One for bios, that takes the value of: undionly.kpxe, one for 64 bit uefi: ipxe.efi, and one for 32 bit efi: i386/ipxe.efi and maybe one for ARM processors. There should be a field for next server or boot server IP. That will be the IP address of your fog server.

I seem to recall a section on tftp, that section is not used for net booting.

@totoro said in JLT 6012 rev 2023 - restor boot:

Same problem with snp.

Ok this kind of points to ipxe being the beginning of the problem. So lets try to update iPXE to the latest release using this thread: https://forums.fogproject.org/topic/15826/updating-compiling-the-latest-version-of-ipxe?_=1694448083008

The logic here is that FOG 1.5.10 was released in March 2023, and you are using 2023 hardware. Its possible that the latest version of iPXE might address this issue. Understand tat iPXE is managed by a different group than FOG, so we rely on the iPXE folks to manage pxe booting issues.

i’d like to look into the network issues with FOS linux, but that should come after we get pxe booting into FOS Linux corrected.

The last thought is that its possible that iPXE will not work on this hardware since it is specialized. We do have a method to usb boot right into FOS linux for those systems where pxe booting is not an option.

@totoro OK good on the latest version of fog.

Just to make sure I can understand. You can capture an image from this device, but you can’t deploy it because ipxe only sends 26% of the FOS linux image?

Is this device uefi or bios based? If its uefi, what boot loader are you using? ipxe.efi or snponly.efi? If you are using ipxe.efi try the snp.efi or snponly.efi. (the snp.efi will use all interfaces, and snponly will only use the interface it pxe booted from) to chain load linux. The snp driver is built into the nic adapter, where ipxe.efi is much like the linux kernel that has all common network drivers built in. That will lead to the next part.

Another observation is that these device are 2023 models == new hardware. Linux sometimes is a bit slow to take up new hardware. As well as new hardware firmware (uefi/bios) sometimes has bugs in it.

@totoro What version of FOG are you using? I find it strange tat you can capture an image, but then when you restore the image the kernel only transfers 26% and fails.

This sounds like an iPXE issue where it is not sending the FOS image complete to the target computer.

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

Sadly USB boot is not suitable for my use case

You are not getting to the FOG iPXE menu, this is an iPXE issue or as I mentioned before your pxe booting is getting hijacked by that WDS suspicious server. Your firmware doesn’t really indicate what is wrong, it just reboots. So that’s not really helpful in debugging.

You could load ipxe onto a usb stick (yes I know you said that USB wasn’t an option) to see if ipxe will boot from the usb drive for testing. If it does then you know the problem is somewhere along the pxe booting chain and not ipxe itself.

I don’t have the pcap anymore, but looking over the thread, I see this WDS server is responding with a bios boot file (wdsnbp.com) that, if sent to the uefi computer, would cause a uefi computer to download it and just reboot. I wanted to see if the DHCP DISCOVER from the client was saying it was a uefi computer, and then this rouge dhcp server was responding with a bios boot loader. That would be strange but also cause the symptoms.

@sgilbe ok so if we map this out we don’t know then if its a problem with ipxe or the FOS Linux kernel. Since other distros boot it might not be the linux kernel. The only caveat is the other distros have older kernels in the 5.x series.

This next step is to build a fos linux usb boot drive. The basis of the build is this tutorial. https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image

Make sure you read the entire thread so you are aware of the caveats. When you get the boot image built, boot into the Grub menu and select the debug option. The idea is to test booting into FOS linux. If it boots then the issue is with iPXE, if it doesn’t boot then we have a linux issue.

Look at the fog forum chat for a few more hints on building this boot image.

Note to myself: Linux hangs on 4th Gen Xeon Scalable processors

https://www.intel.com/content/www/us/en/products/details/servers/multi-node-server-systems/server-d50dnp.html

I know I’ve seen posts in the forum mentioning booting issues with intel scalable processor line.

@sgilbe Can you live boot a standard linux distro like ubuntu/debian/linux mint?

@sgilbe Set your logging level to 7 in the global FOG configuration. Lets see if that gives a bit more detail to what is going on. With logging level at 1 (default) none of the booting messages are displayed.

Just for reference what hardware are you trying to boot FOS Linux on?

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

the machine I am trying to network boot is a Zotac Zbox

This device may simply not be compatible with iPXE. We can get you setup to usb boot into FOS linux to capture/deploy to this device if iPXE will not work.

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

so you are telling me there is already another instance in the network trying/stealing my network boot

In a word, yes. The problem comes with two different sets of pxe boot instructions sent to your computer, it depends on which offer packet get to the target computer first, is the one it will follow. So like 18 out of 20 pxe boots (made up number to explain a point) dnsmasq may respond first, but the other 2 times this WDS service will respond first making your boot into FOG fail. Its something to look into but as you said, is probably not your problem pxe booting that device.