@srcauley If you want to debug the next bit I can help with that too. I’m going to suspect its the nic driver, nic firmware, or raid/disk controller issue. If you setup a debug capture/deploy we can then investigate the errors.

ip a s
lspci -nn | grep -i net
lsblk
grep -i firm /var/syslog

Will get you started

@Ceregon said in Group Kernel Arguments not applied to Host after adding Host to Group while Full Registration:

I had the fear, that when we will have machines without software-raid that will make problems.

That setting just enables the possibility of software raid in the kernel, you still need to use the mdm utility to configure it.

@Ceregon Under FOG Configuration -> FOG Settings, press the expand all button and then search for kernel. In there search for kernel parameters. Add the kernel parameter there. It will be applied to all hosts and all boot methods. For this specific parameter its OK to have it enabled on all hosts, even ones that don’t use software raid.

@srcauley Well I found someone with the same issue as you from last summer, but it looks like the poster ghosted us https://forums.fogproject.org/topic/16922/hand-off-to-fos-kernel-fails-on-certain-gen4-xeon-sapphire-rapids-based-systems-dell-r760-supermicro-x13-etc

This is the one I was thinking of: https://forums.fogproject.org/topic/16993/client-hangs-at-efi-stub it did have a solution with the same error message as you.

@srcauley To quickly explain where its failing.

If you can get to the FOG iPXE menu then undionly/snp drivers are doing what they should. After you pick a menu item from the fog iPXE menu then FOS linux is transferred to the target computer. This is in the form of the kernel (bzImage) and the virtual hard drive (init.xz). The issue is with the linux kernel failing to start up.

Issue #1 that you have is FOS’ kernel is really targeted to workstation class machines, not servers. Servers have “special” hardware that requires drivers that might not be in the FOS kernel. Saying that I know others have imaged Dell servers with FOG.

Issue #2 The R760s are pretty new hardware. You have to remember that linux kernel lags behind bleeding edge hardware. The actual linux kernel might not have the required brains to be able to init this new hardware.

So what should you do?

The first thing is make sure your firmware is up to date (always first step)

Second thing is make sure you are running the latest FOS Linux kernel from the UI FOG Configuration -> Kernel update. Run the latest version 6 release there is.

Third go into FOG Configuration -> FOG Settings, hit the expand all button and then search for log set the kernel logging level to 7 that will send out the most info during the kernel startup. The default is logging level 1 which masks all but the most critical of errors.

Now pxe boot the target computer, see if there is any other info we can glean from the kernel startup.

Also test a linux live boot image, either debian or ubuntu. See if you can live boot the server into linux.

I know there was some debugging in the fog forum with a server that had one of those new scale processors. I think they were able to get that running.

@rodrigoantunes said in Different bios files for different network cards:

wouldn’t it be a dhcp server functionality rather than fog

FOG could do it with some programming, but its probably easier to do it on the dhcp server side. Its not super simple to do, but basically you would setup dhcp reservations or uniqueness based on the mac address of the target computer. When the dhcp server sees a known mac address then it sends out the alternate pxe boot loader. You would make the default pxe boot loader (like undionly.kpxe) be the default and then override it with reservation settings for the troubled computers. It can be done.

I find its strange that the undi driver doesn’t work on some computers but does on others. The undi specification has been around for 30 years. Most firmware has had time to work out the bugs.

@Ceregon said in Problem Capturing right Host Primary Disk with INTEL VROC RAID1:

Maybe i can work with groups of hosts to decide when this postinit script is used

It is possible to access FOG system variables in a pre install script (where I would recommend you do any initialization needed for mdm arrays). So you could use like fog user field 1 or 2 (in the host definition) to contain something that indicates you need to init the mdm array. You might also be able to key it off of the target device name too. I have not looked that deep into it, but it should be possible at least on the surface.

@collins91 I don’t have an answer for you, but I can connect what you see to how FOG works.

With FOG, the FOS Engine (OS that gets loaded onto the target computer for capturing and deploying images) uses NFS to move disk information between the FOG server and target computer.

On the fog server there are two NFS shares.

1. /images that is shared as read only. That is where all of the captured image files end up. The read only attribute keeps the image files from being messed with (i.e. deleted) over the network once they are captured.
2. /images/dev that is shared as read/write. That is where the files temporarily stored at while being captured.

So now you have the basis of the design.
At capture the FOS engine creates a temporary directory in /images/dev share as the mac name of the target computer for uniqueness.

Once the NFS upload is completed. The FOS engine connects to the FOG server using the ftp protocol using the fogproject user account and instructs the OS to move the file pointer for the image from the temp location on /images/dev/<mac_name> to /images/<image_name>. Since only the file pointers are updated this “move” happens very fast.

Then the target computer reboots.

Now to tie this back into what you are seeing, the process seems to be failing on the ftp login and use of the mv command. Your condition kind of tells me the problem is a file permission issue where the (linux user) fogproject doesn’t have the proper permissions needed to execute the mv command.

I would start there for trying to understand why its failed, for example lets say your dead link folder was created by root, and the fogproject user doesn’t have the rights to change a directory created by root. That would cause the process to fail as you outlined.

@mashina said in Providing installation media via pxe booting for UEFI systems.:

Interestingly, the problem doesn’t occur when Ubuntu is already present, and then Windows is deployed. Anyway, that is not a big problem at this moment.

But in this case the uefi firmware has already registered ubuntu as a bootable OS. So it just goes, oh hello I see you again on disk1. But if the entry doesn’t exist then it needs to be fixed up. You might be able to test this on a working system, go into the uefi firmware and delete the entry for ubuntu on the second disk, only leaving windows in the uefi boot manager. Upon reboot does it need to fix itself up again?

Just be aware that FOG doesn’t touch the uefi firmware or boot manager. BUT you can do that with in a FOG post install script and using the linux uefi manager (not the actual name) app. You can add remove uefi boot manager entries at your need.

Your suggestion works well for putting Linux on Disk1, but if the user needs to reinstall Windows, it’ll also go to /dev/nvme1n1, messing everything up

True it will mess everything up. But also I took your inital post as you will load windows once and then could potentially reload ubuntu or the OS on the second drive multiple times. If you “had to” you could write a FOG preinstall script to ask what drive do you want to send the image too, but that gets a bit messy, but its possible.

@pilgrimage This problem is connected to your other post: https://forums.fogproject.org/topic/17255/about-dhcp-and-pxe-problem?_=1707994213348

Your issue is infrastructure and not a fog specific issue. DHCP works based on broadcast messages. These broadcast messages typically are not passed between subnets. On your subnet router there is a service that can be turned on to relay dhcp communications between subnets. I would also think this service is configured on your network because you have client computers on one subnet and servers on another.

Since you are using dnsmasq to provide the pxe boot information since your dhcp server can’t be changed, you need to update this dhcp-helper / dhcp relay agent on your network router to include your dnsmasq server in the list of dhcp servers this service notifies of a dhcp request. Once that is done your dnsmasq server will hear the dhcp request from the remote subnet and respond with the pxe boot information.

@Orfeous said in FOG Project instead of CloneZilla Lite Server:

My goal here is to install Debian or Ubuntu on a PC to be run as a Server. I have a couple of NUCs that I want o deploy an image to via isolated network. Server and Client machines connected to the same switch. No router or such in play.

You can do this on an isolated network completely or install 2 nics in your FOG server and have one connected to your imaging network and one to your business network for remote management on the fog server.

You can also set this up on your business network without interfering with your business network communications. So it can work either way. In some instances you might need access to your business network for AD integration as your target computers boot during its first boot. I understand your goal is linux so AD is not required. But the point is either way FOG will work.

I want this Server to run a DHCP server and broadcast ips to the client machines that will be netbooting via PXE.

If you want to run on an isolated imaging network, just pick to include the FOG DHCP server and the installer script will install the dhcp server and configure it for you.

I want to use those NUCs to boot via PXE and then automatically disk will be restored from image.

If i get other PC vendors and models I want to use another image for those.

No problem on multiple vendors. You just need to really be mindful of the firmware on the target computer bios or uefi modes because the target image is handled a little bit differently between the two firmware classes. FWIW you can not deploy a bios computer captured image to a uefi based computer. The same holds true in reverse.

Is it possible to use my CloneZilla disk image that has already been saved?

While Clonezilla and FOG both use partclone to capture the disk image, the images are stored and compressed differently on either platform. So you can not share the images between the two environments. You will need to capture with FOG if you want to deploy with FOG, or capture with Clonezilla if you want to deploy with clonezilla.

Client NUCs uses NVme ssd and Windows 10 or 11 is located on the disk image.

Now you introduced Windows 10 into the picture. No problem, but that also might mean needing AD during firstboot. You have to remember that the FOS engine (the OS that boots on the target computer) is linux based. So nvme drives have a different disk label that sata drives. But you can capture from a sata drive and deploy to an nvme drive, but that is not a common situation.

Is this possible with FOG Project?

Yes it is

@mashina said in Providing installation media via pxe booting for UEFI systems.:

I encounter a “Reset System” message and the machine reboots

I can say I’ve never seen this before. When you see this message what is the default boot device? Is it the hard drive or PXE booting. Its not clear if its the UEFI firmware or iPXE/rEFInd (fog’s boot loader) that is throwing this error.

This is only a guess, but I might think it the uefi firmware saying “hey something changed from the last boot” should I fix up the boot settings with the new information?

It would be interesting to know if you reimaged the target computer again with the same FOG image would you also get that message, or do you only get the message when you change image formats (i.e. Windows was previously the boot image and now its ubuntu?)

@jmcooper The fault probably exists with the PXE boot rom on the device.

The answer is probably a bit complicated to explain so I’ll try to keep it simple.

PXE booting can use one of the two protocols bootp or dhcp. Its up to the hardware manufacturer to decide which set of fields it looks at for the pxe boot information.

The second part is kind of a dhcp server / pxe rom issue. On each dhcp/or bootp fields there is a byte counter that tells the pxe rom how many characters are in the field name. Common convention also marks the end of a text string with a null (0x00) character. Its up to the dhcp server and pxe rom programmer to decide which method to use (I’m sure there is an RFC document that describes this). I have personally seen this in which someone was not using a mainstream dhcp server, where it only populated the byte counter field and then never terminated the string with a null character. In this case the pxe rom used the null character as end of string and the name requested from the target computer was ipxe.kpxe<with extra characters at the end until a null byte was found>

While I can explain it, it still sucks that the pxe rom firmware is doing this to you.

@RobertD said in PXE boot - HP Elitebook 650 G10 - No configuration methods succeeded ...:

We have been using ipxe.efi for all of our devices for years so I’m afraid to change all our DHCP scopes to use this loader for this one problematic model. Any suggestions?

Its great that you have a path to boot these system via the snp driver.

You didn’t mention if you updated iPXE to the latest release to see if the ipxe.efi boot loader works once again.

Let me (re)clarify in this thread the differences between ipxe.efi and the snpX.efi boot loaders.

ipxe.efi (uefi) and ipxe.kpxe (bios) contain all of the known drivers built into the boot loader, this makes the iPXE boot loader much larger in file size (in 1990 terms of file size) because it has to carry all known drivers onboard the boot loader. For older systems > 6 years those were the preferred boot loaders for iPXE

snp.efi, snponly.efi (uefi) and undionly.kpxe (bios) use the network adapters built in driver through the generic snp or undi interface. This boot loaders are much smaller than the ipxe.* versions since they only need to have one driver onboard (snp or undi). The undi driver (bios) has been around for 30 years and is the preferred and very stable network interface for bios computers and should always be the #1 choice. The uefi firmware has only been around for 12 years or so. The early version were very buggy so the snp driver did not work well. This is the reason why the fog developers recommended the ipxe.efi boot loader for uefi systems. In the last 6 years or so the snp and uefi firmware has matured to a level where the fog developers are recommending snp.efi or snponly.efi for all modern hardware. For bleeding edge hardware you have a better chance to get the snpX.efi bootloaders to work before the ipxe.efi bootloader, because the iPXE kernel developers will need to add the driver to the bootloader. Its just a timing issue.

Now you might ask what is the difference between the snp.efi and snponly.efi drivers. snponly.efi will only initialize a network interface from where it was loaded from. For example lets say you pxe booting a compute with 4 nics, and you pxe bootin from nic2. The iPXE boot loader would only init nic2 using the snponly.efi bootloader. In contract snp.efi would try to init all nic intefaces starting with 1, 2, 3, and then 4. The issue becomes if nic1 takes the boot loader someplace else other than to the FOG server, since the fog server is on nic2.

@alexpolytech94 The shrinking of the disk when using single disk resizable is a bit of black magic. Sometimes because of the actual data size or location of the data on the disk its not possible to shrink the volume down enough to make it fit on a disk that is 1/2 the size of the source disk. I won’t go into too much detail, but if you have a partition that is fixed in size that can’t shrink, but a partition just before it on the disk that can shrink, fog will shrink the one that can be shrunk but leaves the one that can’t be shrunk as it were. If you were to deploy that image to a computer with half the size that non shrunk partition would be technically beyond the last sector on the 1/2 size disk.

To put it another way, always build your mother image on the smallest disk possible, because it can expand to a larger target disk more often than shrink your image to fit on a smaller disk.

When I was building golden images I would build them on a VM with a 50GB hard drive (smaller than anything I would deploy it to) and then let FOG expand the disk to match the target disk size. That always worked.

@zaboxmaster Well we are going to need your assistance to help debug this. I can give you the steps needed to collect the information to help debug this.

1. Schedule a new deploy/capture task, but before you tick the schedule task button, tick the debug checkbox. Now schedule the task.
2. PXE boot the target computer. After several screens of text you need to clear with the enter key, you will be dropped to the FOS Linux command prompt.
3. This next 3 steps will help with remote debugging and for copying the results out to post here. Key in ip a s and collect the IP address of the target computer.
4. Reset root’s password with passwd Give it a simple password such as hello This password will be reset at the next reboot.
5. Now connect to the target computer using ssh, either ssh from linux or putty from a windows computer.
6. Key in and post the results here of the following commands
   uname -a
lsblk
lspci -nn
grep -i -e firm /var/log/syslog
7. Keep the debugging session open for additional commands.

@Dominique said in PXE boot - HP Elitebook 650 G10 - No configuration methods succeeded ...:

I can build other laptops on that network jack
No new setup, have build hundreds of devices a day.

Ok just trying to establish what the problem isn’t. Ok so you updated your ipxe binaries to the latest version. So that’s good.

Looking at the model this hardware is pretty new (Aug 2023). Are you using the nic built into the chassis of the laptop or an external nic in a dock or usbX attached dongle?

Check to see if there is a firmware update available for this hardware. The snp.efi and snponly.efi boot loaders use the nic driver built into the UEFI firmware. On most modern uefi implementations its very robust and stable and should work right out of the box.

I would try for testing purposes to update dhcp to send ipxe.efi boot loader instead of the snp boot loader. Does that boot loader work? The ipxe.efi boot loader is a lot like the linux kernel where it has most of the common network drivers built into the boot loader.

@Dominique I guess I have a few questions in regards to this

1. Is it just this specific model or all models from this network jack?
2. Is this a new setup with FOG or have you used it successfully before?

From your picture I can see that you have the dhcp server setup correctly, you have secure boot off (or ipxe won’t boot), using snponly.efi will use the built-in driver that is part of the nic. You might try ipxe.efi to see if you have better luck, but snponly.efi or snp.efi is probably the best general purpose boot loader.

The no configuration methods screen is basically saying the ipxe kernel is not getting a dhcp response. I have seen this before if you have standard spanning tree enabled on your building switch. In this case you can check to see if its spanning tree issue, but placing a dumb unmanaged / cheap-o switch (like monoprice 5 port switch for $20) between the building switch and the pxe booting computer. If that solves the problem then its spanning tree causing the issue. Enable one of the fast protocols on your building switch like portfast, fastSTP, RSTP, MSTP or whatever your switch manufacture calls it.

@professorb24 I have a tutorial on how to do this with Dell computers, but the concept is almost the same for other vendors. The key is to have the driver files in inf format so pnputil.exe can install them. Lenovo use to have drivers in .exe format and not .inf format. That made things a bit harder when I was trying to setup lenovo computers.

Looking at the script and it doesn’t appear to do anything. other than mount a wim image, but also be aware that the post install script runs withing a linux environment so it can’t directly interact with windows or the .wim file.

Your post install script should copy the drivers over to the target computer after the image is placed on the target computer. All that FOG can do is leave files behind for windows to find them during OOBE/WinSetup.

@rpeterson I have never personally done this, but in theory it should be possible.

As I see it you will have 3 challenges.

1. your uefi boot firmware will need to use the serial port for communications. If you can interact with the computer’s firmware over serial then that clears the first challenge.
2. The ipxe boot loader needs to support a serial console. I don’t know off the top of my head is that feature turned on or not. If its not we/you can make it happen.
3. Lastly (and hopefully the easiest) we will need to turn on the serial console flags for the FOS Linux kernel. FOS Linux is the engine that captures and deploys images. You just need to google “serial console linux kernel” for the proper command line switches.

Once you achieve all of the challenges then it should be possible to do over just the serial port.