Posts made by george1421

@skyborne98 There as been many, many changes since 1.2.x series. One notable change is that the service account fog has changed to fogproject, but that would only come into play during a capture.

The permission denied message during a deploy is suspicious since FOS linux (at this point in a deployment) is running as root. It should have access to the nfs share.

Also for reference, the .fogsettings file is only referenced during an install.

Lets have you run through this tutorial. Remember the service account for FOG is now called fogproject since there were bad instructions on the internet for people to use the fog account to install fog, which reset its passwod. https://forums.fogproject.org/topic/11203/resyncing-fog-s-service-account-password?_=1694461880173 Lets make sure your fog service account is all synchronized before we dig deeper.

@sgilbe I can say I have not tested this process with Rocky Linux (any version). I did create the image for the tutorial using ubuntu/debian version. This issue may be related to a newer version of grub. You might want to try to add the --force switch to the command.

Did you look at the forum chat at the top of the page for additional hints?

@elchapulin This should work for pxe booting except the 32 bit field is bad. It should be i386/ipxe.efi but that probably isn’t your problem since 32 bit uefi systems are rare.

So when you say the pxe client can’t boot, what errors do you see?
Is the fog server, dhcp server and pxe booting client on the same subnet? If yes then we should grab a pcap of the pxe booting process from the fog server using: https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue?_=1694459465428 You can look at the output with wireshark.

In the DISCOVER statement in dhcp options 93 or 94 the pxe booting computer will tell its arch. The dhcp server will take that statement and then (should) seen the proper boot file info in dhcp options 67 as well as the boot-file field in the ethernet header of the OFFER statement. The OFFER statement comes from the dhcp server. If you have only pfsense as your dhcp server then you should have only one OFFER statement, if you see more than one you have multiple dhcp servers on your network.

If you can’t figure it out with wireshark, upload the pcap to a file share site and post the link here or DM me the link with FOG forum chat. I’ll take a look at it.

Also one additional comment, if you have uefi system make sure secure boot is turned off or the firmware will reject ipxe from loading.

@elchapulin Yes it does. (saying these instructions from memory) in the dhcp server there is a section under advanced for netbooting (its not called pxe booting). There was 4 or 5 fields. One for bios, that takes the value of: undionly.kpxe, one for 64 bit uefi: ipxe.efi, and one for 32 bit efi: i386/ipxe.efi and maybe one for ARM processors. There should be a field for next server or boot server IP. That will be the IP address of your fog server.

I seem to recall a section on tftp, that section is not used for net booting.

@totoro said in JLT 6012 rev 2023 - restor boot:

Same problem with snp.

Ok this kind of points to ipxe being the beginning of the problem. So lets try to update iPXE to the latest release using this thread: https://forums.fogproject.org/topic/15826/updating-compiling-the-latest-version-of-ipxe?_=1694448083008

The logic here is that FOG 1.5.10 was released in March 2023, and you are using 2023 hardware. Its possible that the latest version of iPXE might address this issue. Understand tat iPXE is managed by a different group than FOG, so we rely on the iPXE folks to manage pxe booting issues.

i’d like to look into the network issues with FOS linux, but that should come after we get pxe booting into FOS Linux corrected.

The last thought is that its possible that iPXE will not work on this hardware since it is specialized. We do have a method to usb boot right into FOS linux for those systems where pxe booting is not an option.

@totoro OK good on the latest version of fog.

Just to make sure I can understand. You can capture an image from this device, but you can’t deploy it because ipxe only sends 26% of the FOS linux image?

Is this device uefi or bios based? If its uefi, what boot loader are you using? ipxe.efi or snponly.efi? If you are using ipxe.efi try the snp.efi or snponly.efi. (the snp.efi will use all interfaces, and snponly will only use the interface it pxe booted from) to chain load linux. The snp driver is built into the nic adapter, where ipxe.efi is much like the linux kernel that has all common network drivers built in. That will lead to the next part.

Another observation is that these device are 2023 models == new hardware. Linux sometimes is a bit slow to take up new hardware. As well as new hardware firmware (uefi/bios) sometimes has bugs in it.

@totoro What version of FOG are you using? I find it strange tat you can capture an image, but then when you restore the image the kernel only transfers 26% and fails.

This sounds like an iPXE issue where it is not sending the FOS image complete to the target computer.

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

Sadly USB boot is not suitable for my use case

You are not getting to the FOG iPXE menu, this is an iPXE issue or as I mentioned before your pxe booting is getting hijacked by that WDS suspicious server. Your firmware doesn’t really indicate what is wrong, it just reboots. So that’s not really helpful in debugging.

You could load ipxe onto a usb stick (yes I know you said that USB wasn’t an option) to see if ipxe will boot from the usb drive for testing. If it does then you know the problem is somewhere along the pxe booting chain and not ipxe itself.

I don’t have the pcap anymore, but looking over the thread, I see this WDS server is responding with a bios boot file (wdsnbp.com) that, if sent to the uefi computer, would cause a uefi computer to download it and just reboot. I wanted to see if the DHCP DISCOVER from the client was saying it was a uefi computer, and then this rouge dhcp server was responding with a bios boot loader. That would be strange but also cause the symptoms.

@sgilbe ok so if we map this out we don’t know then if its a problem with ipxe or the FOS Linux kernel. Since other distros boot it might not be the linux kernel. The only caveat is the other distros have older kernels in the 5.x series.

This next step is to build a fos linux usb boot drive. The basis of the build is this tutorial. https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image

Make sure you read the entire thread so you are aware of the caveats. When you get the boot image built, boot into the Grub menu and select the debug option. The idea is to test booting into FOS linux. If it boots then the issue is with iPXE, if it doesn’t boot then we have a linux issue.

Look at the fog forum chat for a few more hints on building this boot image.

Note to myself: Linux hangs on 4th Gen Xeon Scalable processors

https://www.intel.com/content/www/us/en/products/details/servers/multi-node-server-systems/server-d50dnp.html

I know I’ve seen posts in the forum mentioning booting issues with intel scalable processor line.

@sgilbe Can you live boot a standard linux distro like ubuntu/debian/linux mint?

@sgilbe Set your logging level to 7 in the global FOG configuration. Lets see if that gives a bit more detail to what is going on. With logging level at 1 (default) none of the booting messages are displayed.

Just for reference what hardware are you trying to boot FOS Linux on?

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

the machine I am trying to network boot is a Zotac Zbox

This device may simply not be compatible with iPXE. We can get you setup to usb boot into FOS linux to capture/deploy to this device if iPXE will not work.

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

so you are telling me there is already another instance in the network trying/stealing my network boot

In a word, yes. The problem comes with two different sets of pxe boot instructions sent to your computer, it depends on which offer packet get to the target computer first, is the one it will follow. So like 18 out of 20 pxe boots (made up number to explain a point) dnsmasq may respond first, but the other 2 times this WDS service will respond first making your boot into FOG fail. Its something to look into but as you said, is probably not your problem pxe booting that device.

@gabrielzeit Remember dnsmasq configured this way is also a “dhcp server”

If we look at the packet from the .205 dhcp server, you will see the next server points to .113 (assume to be wds server) and the boot file is surely a wds boot loader (akin to undionly.kpxe). The .1 dhcp server (looks like maybe a router) is also telling the same story.

@gabrielzeit I have to run to a meeting but your pcap is interesting in that you have 3 dhcp servers and 2 of them are pointing at a WDS deployment server.

.201 and .1 are pointing to WDS and .49 appears to be the fog server.

This configuration will cause intermittent issues.

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

Then blackscreen. And this repeats every few minutes.

Ensure that secure/safe boot is disabled on the target computer.

@gabrielzeit So just to be clear the device that is hanging with no configuration methods succeeded is a VirtualBox VM and not a physical computer?

If a physical computer is doing this, I would think its spanning tree not something to do with ipxe.

Now if its ipxe then Tom’s recommendation is the better answer. VB is a strange critter as it were. Just be aware that ipxe.pxe is only supported in bios modes and not uefi.

@gabrielzeit Your dnsmasq config is not complete and a bit out of date.

What version of dnsmasq are you running? dnsmasq -v
Here is a tutorial with a good configuration for dnsmasq version 2.75 and later: https://forums.fogproject.org/topic/12796/installing-dnsmasq-on-your-fog-server?_=1693476360006 It will dual boot bios and uefi systems.

@DBCountMan Yes if you are using the location plugin. Using the location plugin you assign the storage node to a location and then when you register the target computer with FOG you assign the target computer to a location. Then when the computer pxe boots it learns who it needs to communicate with. So a storage node would work in this case to spread the load.

One caveat with storage nodes is that they can’t capture images. Only the master node can capture images. Then all storage nodes (including the master node) in the same storage group can deploy the image.

@alexamore90 You have painted an incomplete picture here so I can only speak in general terms.

Using modern target computers you can flood a 1GbE network link with 2-3 simultaneous unicast imaging. Any additional number of simultaneous will impact the overall performance of all systems imaging. On a well designed 1GbE network you should be able to get between 5.5 and 6.5 GB/min as displayed on the partclone screen during imaging. For a 10GbE network you should get between 13 and 15 GB/min. (your performance metrics of 16gbs and 1000 and auto-negotiate are inconsistent). If you only have a 1 GbE network, you can add additional network cards configured in a LAG trunk to spread the imaging load of 10 computers across multiple nic cards.

During imaging the heavy load is carried by the target computer. The FOG server only moves disk blocks from its local storage and sends them out the network adapter. The fog server does monitor the imaging process but does not perform heavy computational load. The speed of the disk subsystem on the FOG server will also have an impact. If your fog server only has 1 physical spinning disk, that disk will have to work hard to keep up with delivering 10 data streams from different locations on the disk platter. In this case if you only have spinning disks, setting up a striped raid configuration using multiple disks will help or moving to SSD drives.

The last comment is if you need to image 10 systems at a time, you might look into FOG multicasting mode. This mode requires some setup and a network infrastructure that supports multicasting, but with this mode you can image 30 or more systems at the same network bandwidth as just 2 multicast streams.