@robertkwild Yes right now pxe booting issue is between uefi firmware and network adapter. Typically the hardware vendor will have a recommended usb adapter (typically hardware vendor branded $$) that supports pxe booting, where the driver for the usb adapter is built into uefi.

I the imaging process once bzImage is loaded (FOS kernel) and you have a network issue then the issue with with the “FOG kernel”. You are not there yet in the booting process.

@jmeyer said in Error PXE-E18 - Lenovo ThinkPad E16:

I’ll put a wireshark tomorow to look what happen.

You really need to see what the client is being told and by who to try to explain why bios works and uefi doesn’t. There is something unknown going on here.

@robertkwild You have two things at play here.

1. The laptops don’t have built in nics so you can pxe boot them.
2. If you don’t purchase usb nic drivers that is supported by the laptop vendor they will not pxe boot.

You have to consider that uefi bios is much like a linux kernel. The uefi firmware vendor has control over what usb nics they support for pxe booting. If the laptop vendor doesn’t support the nic you will not be able to pxe boot. This isn’t a fog issue, its between the laptop vendor and the nic vendor.

The second issue is when ipxe boots it must have the correct nic driver onboard to boot.

The third issue bzImage boots that is the FOS Linux kernel. That is in the realm of fog developers. If the driver is available for linux, it can be included in bzImage. You will have better luck if the usb nic is a bit older and a bit more generic and linux will probably have a driver.

Now if you have a usb nic that is supported by linux but not the vendor, you can always usb boot right into FOS linux (bzImage) and bypass pxe booting altogether. This won’t work in a campus environment but will work on the imaging bench. I think I have a tutorial to usb flash drive into iPXE to get more normal imaging experience. Booting right into bzImage has a few caveats.

@davidsontiago If you are using sysprep, include the command line option to tell sysprep to power off the computer. You have this message because when you pick shutdown from the menu, windows doesn’t actually shutdown the computer but put it into a sleep state leaving some files open that FOG sees. Sysprep will power off the computer correctly for cloning or use the following command “shutdown -s -t 0” to properly power off the source computer and close all open files.

@jmeyer ok you did a lot of the initial debugging so its not the easy stuff.

The bit harder stuff is using either tcpdump on the fog server if its on the same subnet as the pxe booting computer. Or a witness computer loaded with wireshark on the target computers subnet.

If the fog server and pxe booting computer are on the same subnet you can use this tutorial to capture a pcap file: https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue you can look at the pcap with wireshark or post it here and I will look at it.

or if your target computer is on a different subnet than the fog server you will need to use a witness computer with wireshark loaded. Use a capture filter of “port 67 or port 68” to only grab the DORA (dhcp) packets on the network.

What you want to look for is there will be a discover packet from the target computer that says “hello I’m here, come configure me”.

You will get a OFFER from one or more dhcp servers. This is the packet you are interested in. In the OFFER there will be a header section with the fields next-server and boot-file. These need to be populated with the fog servers IP address and snponly.efi (or whatever). This is the bootp fields. In addition if you look down in the dhcp options 66 and 67 that should match what is in next-server and boot-file fields. I’m suspecting something is not right with this section.

This is a pxe boot (iPXE) issue and unrelated to what FOS kernel or version of FOG you are running.

@DBCountMan Its been several years since I last messed with it, but if I remember correctly we needed the nsfv4 components compiled into FOS OS. I have the tools to do that for the testing, but I thought those changes were merged into the inits, but I’m not sure without looking. They can and should be there since there is zero impact to nfs v3.

You are right its a lot of I tried this and it worked instead of a clear tutorial. If I can make some time I should try to write up a more formal how to.

@DBCountMan Its not officially supported but in the feature request I put in so many years ago I had it working.

@lperoma Or you can leave fast startup enabled and either have sysprep power off the computer or use the shutdown command to properly close the file system for imaging. I think the shutdown command is shutdown.exe -s -t 0 to properly closed all files.

@kbats183 I haven’t looked at the quick registration bit of code but I did have a hack that updated the fog.auto.reg (full registration) option to begin imaging right after registration without a reboot. This involved taking the FOG supplied script updating it to your requirements and then adding a bit of code onto the end and then finally dynamically patching that script on every boot of the target system. It sounds like a lot of steps and its complicated but not if you know how fog works.

I should be able to point you in a direction so you can get started.

When FOG Linux (the os that gets transferred to the target computer for imaging) boots it runs the master fog script stored in the /bin directory in FOS Linux. The scripts bits of that /bin directory is here: https://github.com/FOGProject/fos/tree/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin The file you are interested in is called fog.auto.reg. If you can understand bash script programming you can modify this file to fit your requirements. Once you have the script the way you need it then you can dynamically patch when FOS linux boots using this tutorial (understand this tutorial is for patching fog.man.reg but the concenpt is the same only the file name changes.

https://forums.fogproject.org/topic/13500/dynamically-patching-fos-using-a-postinit-script

I have tutorials that are targeting the manual registration but the concepts are similar. (hint: this is where I tweak the script to give me a custom calculated host name kind of like the auto naming but better. )
https://forums.fogproject.org/topic/14278/creating-custom-hostname-default-for-fog-man-reg

So that’s the background, how do I make it image right after rebooting?
For the fog.man.reg file you append a bit of code that makes the script think the system rebooted and collects the latest imaging info.
(For the life of me I can’t find that tutorial I created but here is the script that I collected from a recent issue with the code in the script.

    sysuuid=$(dmidecode -s system-uuid)
    sysuuid=${sysuuid,,}switch
    mac=$(getMACAddresses)
    base64mac=$(echo $mac | base64)
    token=$(curl -Lks --data "mac=$base64mac" "${web}status/hostgetkey.php")
    curl -Lks -o /tmp/hinfo.txt --data "sysuuid=${sysuuid}&mac=$mac&hosttoken=${token}" "${web}service/hostinfo.php" -A ''
    curl -Lks -o /tmp/hinfo.txt --data "sysuuid=${sysuuid}&mac=$mac" "${web}service/hostinfo.php" -A ''
    [[ -f /tmp/hinfo.txt ]] && . /tmp/hinfo.txt
    . /bin/fog.download 

In the case of fog.man.reg this code would be added at the very end of the script.

Looking at the fog.auto.reg script I would say it should go at the very end of that script too.

ref: https://forums.fogproject.org/topic/17601/deploy-image-right-after-registration-without-a-reboot/3
ref: https://forums.fogproject.org/topic/16378/start-imaging-right-after-the-full-host-registration-without-reboot-possible

@Bhav In addition to what Tom posted, the server response timeout is suspicious too. IF the target computer is in uefi mode and it downloaded undionly.kpxe the target computer should have responded with something about undionly.kpxe is not the right format. But in this case its getting a server response timeout. This makes me think two things. 1) your dhcp opition 66 is set incorrectly or the name of the file has a capitalization issue. For linux Undionly.kpxe and undionly.kpxe are not the same thing. Either way I would start with your dhcp server and make sure dhcp options 66 and 67 are set correctly for a uefi based computer.

@Quintin-Giesbrecht said in FOG Very Slow to Deploy Image - Lenovo Neo 50Q Gen 4:

Lenovo Neo 50Q Gen 4

I’m with Tom here, I would go with the latest FOS Linux kernel with this new hardware. You didn’t happen to mention what version of the FOS Linux kernel you are using (FOG UI -> FOG Configuration -> Kernel Update). I would go with the latest 6.6.x or later kernel. This is the easiest and quickest way to see if it solves the problem. It would be useful to know what version is your current kernel too so if we see this issue again we can help the next guy.

@lperoma Thank you for the pcaps. There is something strange going on here.

This is what I gleaned from our discussion (actors)

192.168.0.40 is dnsmasq
192.168.3.253 is my gateway (and DHCP server).
192.168.0.21 Fog Server
192.168.0.0/255.255.252.0 /23 network

When I look at the pcap the target computer 192.168.3.8 is downloading undionly.kpxe from the dnsmasq server 0.40 and not the fog server 0.21. At no time in the pcap do I see 0.21 in the dialog. Also it looks like the target computer is on a different subnet than the dhcp server and fog server. The pcap only shows one side of the conversation. I can see the Discover and Request bits of DORA. I’m missing what the dhcp server(s) are telling the target computer. This is because the client is on the other side of some router. The target computer is indeed a bios base system (HP) and the 0.40 server is running as a VM under proxmox.

So there are several questions here.

1. Is your fog server really 192.168.0.21
2. Why does the dnsmasq server have FOG’s pxe boot files? Those should be on the fog server.
3. Why does dnsmasq only tell the client computer to boot from itself and not the fog server?
4. You are capable of causing a good pxe boot and not good pxe boot. What are you doing different, or what is different between the two pcaps?

It would be interesting to see the other side (what the client is being told) by taking a third computer loaded with wireshark to the clients subnet and then use a capture filter of port 67 or port 68 to see what actors are involved and what the client is being told. At this time I’m only interested in what is not working.

ref:

@lperoma TBH I never look at the DORA process from the server side because it only tells us half the story. It looks like its working right in both flows.

Are you sure the dnsmasq is configured on the .40 server as a dhcp proxy or is it doing traditional dhcp functions. In a dhcp proxy config it only sends the boot file override command and then at the end of the DORA process it will then answer the dhcp proxy request on port 4011. Your dnsmasq configuration should look similar to the one in this tutorial: https://forums.fogproject.org/topic/12796/installing-dnsmasq-on-your-fog-server

When I try to debug this process I use the process outlined in this document: https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

Since your dnsmasq server isn’t on your FOG server, I would run the tcpdump command from your dnsmasq server since there is some unicast communications to dnsmasq you would miss if you ran it on the fog server. It would be intersting to see what the client is being told on the pxe booting process that fails.

Minus the dhcp proxy communications you would miss you could also use wireshark with the capture filter of port 67 or port 68 running on a witness computer on the same IP subnet as the pxe booting target computer. There may be an unknown dhcp server mucking up the works on this pxe booting.

If you are willing to share the pcap I will look at it to see if there is anything that jumps out at me as the problem.

@LiamRetrams I can’t give you the exact steps to solve this but I can point you in a direction.

You will need to solve this in the windows realm. Once oobe starts FOG imaging is out of the picture so we can’t do anything in a post install script. You will need to use the bcdedit commands from inside windows to reset the boot order. You should be able to do this within the startupcomplete.cmd batch file or using the unattend.xml file in the first run section, or possibly deploy a fog snapin to run a batch file with the bcdedit commands.

I would manually reset the boot order to the way you want it then from within windows use the bcdedit commands to look at how it is ordered via bcdedit. That should give you the structure of what you need to do with the bcdedit commands to rebuild that format on a different system.

@lperoma said in Chainloading failed, hit 's' for the iPXE shell:

192.168.3.253

What device is that (manufacturer and model)? is that a soho router?

How does dnsmasq fit into this design? Is it setup as a proxy dhcp server?

The reason why I ask is I’ve seen many soho routers say (tell client computers) they are the boot server even if you have dhcp option 66 set correctly. The bios on the target computer is getting ipxe loaded correctly. Once iPXE starts it issues a dhcp discovery request again to find the boot server’s IP address. What it is getting this time is your gateway’s address.

With this new information I don’t think its spanning tree.

@lperoma So I can make the generalization statement that time fixes your issue?

What I want you to test is this.

PXE boot into the error and then press s to get to the ipxe console.

Wait 20 seconds.

Now key in the following at the console prompt
chain tftp://192.168.3.253/default.ipxe
Does this work and continue into the FOG iPXE menu. In that the only thing different between not working and working is time? (I’m driving to a point here).

If time fixes the issue, lets place a dumb switch (unmanaged cheap-o like monoprice) switch between the building network switch and the target computer. Now pxe boot the computer does it boot correctly? If yes then its possibly a spanning tree issue with your building network switch. Make sure your building switch has fastSTP, portfast or whatever your switch mfg calls the fast spanning tree protocol. I might be off base here but it sounds like it might be spanning tree. I know that processor is pretty old, but I don’t think its cpu related.

@lperoma Undionly.kpxe and snp.efi are probably the safest choices. Both of these use the built in driver built into the network adapter. The ipxe.pxe and ipxe.efi are more like the linux kernel where it has every known driver built into the boot loader. All of the other ipxe boot loaders are for unique situations that are not common.

So your system is an old 4th gen i5 processor. That’s strange that ipxe is misidentifying that CPU.

@lperoma said in PXE Boot Error - "Kernel Panic" Issue "XZ decompressor ran out of memory":

Any other idea? Can some one identify any issue with my dnsmasq ?

Its not a dnsmasq issue. dnsmasq only instructs to pickup ipxe.efi or undionly.kpxe once that boot loader is running dnsmasq has done its job. For some reason ipxe is misidentifying the processor as 32 bit only. Will you collect the exact processor model in these AIO?

I just saw in your previous message ipxe.kkpxe is being sent. Update your dnsmasq to send undionly.kpxe instead. That shouldn’t matter because ipxe.kkpxe uses the internal network adapters and undionly.kpxe uses the generic UNDI driver. But for consistency sake, lets make sure its not an ipxe issue.

@lperoma I don’t feel this is a pxe booting issue. There is only two flavors of iPXE. One for bios and one for uefi. Is this computer in bios or uefi mode?

The only way I can see this messing up because of dhcp is that the computer is in uefi mode, and you had something wrong with dnsmasq, where its loading the 32bit version of iPXE. That might falsely miss identify the processor as a 32 bit causing bzImage32 to be called.

The other thing I can think in the fog configuration -> fog settings. If you hit the expand all button then search for bzImage. If by chance someone changed the 64 bit fields to load bzImage32 and the 32 bit initrd. But that’s really unlikely.

Do all other computers work correctly except for this one?

@lperoma Looking at your screen shot a few things jump out at me.

Why is iPXE loading the 32 bit version of FOS Linux?

What hardware are you running here? What is the CPU? Is this super old hardware, like from the Pentium era?

I translate this error to the kernel (bzImage32) doesn’t have enough RAM memory to expand the FOS linux virtual hard drive (init_32.xz) into memory. If this IS a 32 bit system I still can’t understand why there is not enough ram for this kernel expansion. 32 bit arch is limited to 2GB of RAM. The FOS linux kernel and initrd compressed is < 400MB. Something is not adding up.