@mbghost I’m still leaning towards init.xz corruption. Its very strange that on a fresh fog sever it works on day one and then the next its no good.

Just for clarification its all and every Toshiba all in ones but other models work just fine? Its only this specific model.

What I’m thinking at the moment is that bzImage transfers fine and its around 8MB in size. The kernel also boots fine because its getting to the point where it attempts to connect to the root file system.

init.xz is a zstd compressed image. Its compressed size is around 350MB. Both images are very small in size. Something is happening to the init.xz image to where bzImage is not able to mount it and the kernel panics.

This image persists across multiple deployments and multiple installs of FOG server. It also crosses different init.xz and bzImage kernels.

FOS linux does boot 1 out of 12 attempts.

So where could the problem hide?

1. The FOG server hardware if that was a consistent deployment throughout the server rebuilds. (test try building fog server on a desktop/laptop computer to rule out fog server infrastructure)
2. Something with the network between the fog server and the target computer. (move target computer as physically close to fog server as possible and test deployments eliminating all of the existing networking between fog server and target computer)
3. Something with the target computer. (if you have been testing with the same computer throughout these tests use a different computer. Its possible there is a ram issue with this computer)

Right now there isn’t a clear picture on the cause. I can say this IS unique and I’ve haven’t seen this before with FOG.

Something else you might do is in the fog settings, set the log level to 7. I think the default is 4. 7 is verbose and the kernel might spit out more information to why its not happy with the init.xz file. Like decompression failed.

@mbghost This error message baffles me. If its happening where I think its happening its not a pxe boot issue. This error happens after you pick an iPXE menu item or if you tell a computer to image.

So you can probably rule out ipxe.efi/snp.efi here.

This error message is generated with FOS linux is booting. The kernel has booted and when it goes to connect to init.xz the format of init.xz is corrupt for some reason.

What version of FOG are you running
What version of “the kernel” are you running?
What version of init.xz are you running (get this from a bios computer that boots. the version of the init will be under the fog logo)

What computer is this happening on (make and model)?
Is it all uefi systems or only from one manufacture?
How much ram does this computer have?
Are you seeing both bzImage and init.xz get transferred completely to the target machine. This will be visible just after you pick an item on the FOG iPXE menu.

To me this error is telling me something is wrong with init.xz or for some reason bzImage is not the right kernel for init.xz

@mbghost said in Unable to Get IP Address After PXE Menu on Physical PC (FOG Project on ESXi):

ESXi server → Cisco Switch → Client.

So just to be clear pxe boot the vm on esxi works no prob, but physical host does not.

Lets test this, on the target computer, put one of those cheap unmanaged switches (like the $20 monoprice ones) between the pxe booting computer and the building network switch. Now try to pxe boot. If it works then get with your networking group and make sure the switch ports are configured for portfast, because its spanning tree causing you some troubles. Understand this is an educated guess based on what you’ve posted.

Just for some background on this, standard spanning tree takes 27 seconds to start forwarding traffic. FOS Linux boots in under 15 seconds, so its already given up trying to get an IP address by the time spanning tree starts forwarding data.

@uzee FOG doesn’t change or really step into the target system during deployment. You CAN do this if you write a post install script that will get called once the target image is deployed to the target computer. In the case of a linux computer you could potentially set the target computer’s name here by updating the /etc/hostname file or anything else you can do with a bash script. The FOG Client can also be configured to make changes to the target computer or install applications based on instructions from the fog server. Its really identifying what you want to do and how much automation you want to script. Its all possible with FOG and a little creativity on your end. Since you already know Ubuntu (linux) bending FOG to what you want will be easier than coming strictly from a MS Windows background.

@uzee Your approach to this (imaging) really depends on your short term and long term goals.

If imaging these machines are just a one time time (like if you were a PC reseller) then there is a process I call load and go. where you pxe boot one time into the FOG iPXE menu and then issue a Deploy Image from the FOG iPXE menu. This will deploy the target image to the target computer one time and then the FOG server will forget about the computer. Again you would use this method if you were going to stuff an OS onto some hardware and then never see that target computer again.

The other method is the traditional registration, assign an image and then schedule an image deployment. In this case the FOG server will register and know about the target computer in the future. You have to remember though that you register the target computer once and then can deploy many times. Future remote imaging will be possible because you register the target computer with FOG. FOG also gives you the ability to manage the computers in the future if you install the FOG Client.

Now with this registration and then booting. The registration process must be hands on, but imaging can be touch less or at least low touch. Some people will have the target computer configured so that it always pxe boots. In this case the target computer will always pxe boot through the fog ipxe menu. The default action if you don’t touch the computer will be to boot the OS off the hard drive. This boot through the iPXE menu, will also check to see if the fog server has any tasks assigned for the target computer. If there is a task assigned (like reimage the computer) then instead of booting through to the local hard drive, it will boot into imaging. If you couple that with the fog client install on the target coputer’s host OS. When you schedule a deployment to the computer, the fog client will then reboot the computer and then the bios is configured to boot through the fog menu, the target computer will be reimaged without requiring a tech to touch the computer. You would use this process to maybe reset a computer lab between classes, where you can wipe the target computer and then reload the OS on 30 computers in 10 minutes.

The interrupt the booting process twice, is actually pressing F12 to get into the firmware’s boot menu to pick PXE booting. You are not changing the forever boot order, you are just picking one time pxe boot. Yes you will have to do that twice, but this way the default boot is always the hard drive.

@Ryxn I checked and that nic has been in the linux kernel for several years.

And secondly when you checked for missing firmware the kernel wasn’t complaining about missing something.

Its a good thing you were able to get the error while in debug mode. Because that was going to be the next quest for you. To get the system when its behaving badly.

So when you get it into debug mode and its complaining about no nic lets do the following.

ip a s should show if the network interface has an ip address, I’m going to suspect no because it complained about it earlier in the boot process.

lspci -k -nn | more Look through this list until you find the entry. The number in the square brackets will be what you posted earlier. [8086&1502]. See if the nic is visible to the kernel. AND if it is it should mention what kernel driver its using. I’ll give you a for example using my laptop.

0000:00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (
13) I219-V [8086:15fc] (rev 20)
	Subsystem: Dell Ethernet Connection (13) I219-V [1028:0a22]
	Kernel driver in use: e1000e
	Kernel modules: e1000e

lastly if everything looks good, kernel sees the nic, there is a kernel driver assigned. Lets see if time fixes your problem.

When you ran the ip a s command it listed all of the network interfaces and if it has IP addresses. You will need to know the interface name for your network adapter. It may be something like eno1 or ens192, or something else. Get that device name.

Now run this command /sbin/udhcpc -i $iface --now replacing the whole word $iface with the device name listed from the ip command. This will tell the network adapter to again poll for an IP address from your dhcp server. If it picks up an IP address this time, then time does solve your problem and you will need to look into your network configuration and make sure that port-fast or fast-stp is enabled on your network port.

@ProfessorFow OK what I want you to do is on the master node, go into the storage group where this remote node is. Look at this remote node. There should be a ftp user ID and password listed there (said from memory). Use those credentials to connect like you did with the ftp CLI from the master node to the remote node. See if you can login using ftp to the remote node. Then issue a ls command, Its not important what the answer is as long as it gives you an answer and not an error.

If you can’t login using those credentials then on the remote storage node look at the /opt/fog directory. There is a hidden file called .fogsettings. issue the command cat /opt/fog/.fogsettings In that file should be the password for the account used when the fog server was installed. Make sure they match what is on the master node.

There is something with ftp that is not working correctly, that is why the replication is not happening.

@ProfessorFow The FOG replicator uses ftp to transfer files to the storage node. Make sure ftp is not being blocked between the master server and the remote storage node. You might test by on the master node, issue `ftp <remote_server_ip> if you are not prompted to enter a login then FTP is being blocked.

@Ryxn Ok to divide and concur this issue, if you are getting to the FOG iPXE menu then your pxe booting is working ok and that’s not the problem.

Since you mentioned bzImage, and the network interface shutting down once bzImage starts then issue is with FOS Linux kernel, 6.1.22 is your current version. The issue is with this 6.1.22 kernel and nothing else.

What we need is the hardware ID of that network adapter. In windows you can get it from the device manager hardware ID. There will be a vendor code and a device code. Under linux if you run lspci -nn | grep -i -e net the code will be in the form of [8086:1fd6] (made up code). Once we have that code we can look to see if there is a linux driver for that nic.

You need to schedule a task for the target computer and tick the debug checkbox. PXE boot the computer and after several screens of text you will be dropped to a fos linux command prompt.

Also run this command, the file is either /var/log/messages or /var/log/syslog (I can’t remember)

grep -i -e firm /var/log/syslog (remember the file may also be messages)

What we are looking for is the kernel complaining that the nic requires a specific firmware to activate the nic. To keep fos linux fast and light there are only a limited number of firmwares built in. That can be fixed but we need to know which one.

@Donutsrule5

OK lets see if we can boot from a usb drive. This tutorial will instruct you on how to build a usb boot image. https://forums.fogproject.org/post/69606 the google drive link below will take you to step 7 in the thread.

https://drive.google.com/file/d/1psCrPVzBTvlakLkCMvhdoScAEZp1CKE6/view?usp=drive_link

Once you build the usb boot drive, take the kernel and init.xz from your current version of fog and replace the ones on the flash drive with the ones in from the FOG server. The files are in /var/www/html/fog/service/ipxe directory.

Once you have the usb drive ready, pxe boot into the grub menu on the flash drive, then go into debug mode. After several pages of text you need to clear with the enter key you will be dropped at a fos linux command prompt. If you can get that far then fos linux is OK and the problem is probably with ipxe or the handoff between the two.

If you want to image with the flash drive, schedule the task in the fog server BEFORE you usb boot and select option 1 from the menu. If you get it out of sequence you will get an error that says something about Null. Basically its saying you don’t have a task scheduled on the fog server so there is nothing to do. Lets see if we can get it to boot this way.

@sideone said in [BUG] iPXE Boot Loop & Menu Failure After FOG 1.5.10.1660 Upgrade:

Your suggestion of trying a realtek.efi didn’t work

At running the risk of polluting this thread just a tad more. If undionly.kpxe worked in bios mode then snp.efi should work in uefi mode.

Also knowing the exact realtek nic model could be important if its a 8168 (1GbE) or 8125 (2.5GbE) the default drivers in the linux kernel wont work well. You can try to use the experimental kernel that has the official realtek drivers built in https://github.com/rluzuriaga/fos/releases/tag/EXPERIMENTAL_REALTEK_DRIVERS Understand this will only help once you get past the FOG iPXE menu. This would be after undionly/snp has displayed the FOG Menu.

@Donutsrule5 said in Del Pro 14 Premium PXE fails:

Any ideas or help

This is failing at the nexus of iPXE handing off control of the computer to FOS Linux. So it could technically be either iPXE or FOS linux. The version of FOG is mostly irrelevant at this time.

I would say rebuild ipxe and make sure you have the latest FOS Linux kernel, I think 6.12.35 is the latest, but not sure. Here is a tutorial on how to rebuild ipxe: https://forums.fogproject.org/topic/15826/updating-compiling-the-latest-version-of-ipxe There is also a link in that article to the official wiki page with the updated instructions too.

We are not done, at a road block yet. I can create a newer one off kernel based on 6.15.5 as well as we can build a USB boot drive to boot FOS Linux from a USB stick and bypass iPXE and the handoff between the two.

@aurfalien said in Mellanox ConnectX 3 imaging issues.:

So all good here.

Did FOS Linux have the driver enabled for that 10GbE card (i.e. were you able to image with FOG once you got past the iPXE menu)?

FWIW: For uefi mode snp.efi or snponly.efi should work if undionly.kpxe worked in bios mode. I’m saying it should because ConnectX3 is a somewhat older card (circa 2017/2018).

@punch07 to view the /tftpboot use the command line terminal window. Now I do have to say that different distros have the directory in different locations like /var/lib/tftpboot.

@punch07 I agree from the picture everything looks good.

What we don’t know is the number of actors involved with this pxe booting process. We really need to know what the client is being told and by who.

Follow the instructions in this tutorial. Collect the pcap and look at it with wireshark.
https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

What will be specifically interesting will be the OFFER packets. If you only have 1 dhcp server on your network you should have only one OFFER packet. If you have more than 1 then you need to identify each dhcp server that is responding.

If you have only one OFFER packet, then look into that packet. In the ethernet header you should have two fields. {next-server} and {boot-file} those settings should match what you have set in dhcp options 66 and 67. Both the bootp section and the dhcp options section need to be set.

If you can’t figure out the pcap, unload the pcap to a public file share site and paste the link here. We can take a look at the pcap and give you additional guidance on where to look, but in the pcap file you should see at least 4 packets, DISCOVER, OFFER, REQUEST, ACK/NACK. The OFFER packet comes from your dhcp server(s).

One additional note if you have a primary and backup dhcp servers make sure both have the boot settings configured.

EDIT: I just thought of another issue it might be. If you did not complete the installation properly the /tftpboot directory on the fog server might be empty. Double check to see if you have ipxe.efi in that directory. If its not there, make sure you complete all 3 phases of the install. fog server console->web ui to install database-> fog server console to complete the install.

@punch07 said in Unable to install Fog successfully.:

DHCP server is a router or a win server

I guess I need to ask the question a different way. Did you update the configuration on your dhcp server to include dhcp options 66 to the IP address of your fog server and dhcp option 67 to be ipxe.efi? According to the error message its saying your dhcp server is not handing out the boot file name (dhcp option 67).

If your dhcp server is a soho router, then I understand why its not working. If it is a ms windows server then you need to look into the setup.

@punch07 The issue is related to your dhcp server not specifically FOG. Your dhcp server is not sending out the boot file name, that is what the client is complaining about.

What device is your dhcp server for this netwokr?

@bobbywitt Are you having a problem or are you just seeing if others have had issues?

@skyhawk3355 I remember awhile ago I needed to create a one-off kernel for a system that had a 486 cpu. I checked but I don’t have that kernel any more.

I just recompiled the linux 6.6.85 for a 586 based CPU. I can recompile for a 486, but I don’t know if that is going to get us anything better.

Here is 6.6.85 586 version
https://drive.google.com/file/d/1GAzFjbtpDVXXCRpe6bIZ5mwySSXeDHko/view?usp=drive_link

Here is 6.6.85 586 where I stripped out some drivers like scsi, nvme, uncommon network drivers, virtualization drivers, etc. I was able to strip out over 1MB of kernel size from the previous one above. Did I through out too much?? YMMV.
https://drive.google.com/file/d/13dE7BLgofsFiNJj_Q8nLDknplRbWRqkJ/view?usp=drive_link

@Eazis said in ASUS NUC14MNK fos kernel no netwerk drivers:

So the 8125 driver was missing?

Well if you close one eye and squint with the other, you can still see it. The kernel developers consolidated all of the realtek drivers into the 8169 driver which kind of works except the driver updates lags behind the hardware by a few months. It looks like the FOG developers have a way to integrate the official realtek drivers moving forward so that should make life with the reaktek nics easier because realtek releases a new subversion of the hardware every few months.

Anyway I’m glad you have your issue sorted out and the FOG developers have a plan forward too. @rodluz (well done!)