@totoro said in JLT 6012 rev 2023 - restor boot:

Same problem with snp.

Ok this kind of points to ipxe being the beginning of the problem. So lets try to update iPXE to the latest release using this thread: https://forums.fogproject.org/topic/15826/updating-compiling-the-latest-version-of-ipxe?_=1694448083008

The logic here is that FOG 1.5.10 was released in March 2023, and you are using 2023 hardware. Its possible that the latest version of iPXE might address this issue. Understand tat iPXE is managed by a different group than FOG, so we rely on the iPXE folks to manage pxe booting issues.

i’d like to look into the network issues with FOS linux, but that should come after we get pxe booting into FOS Linux corrected.

The last thought is that its possible that iPXE will not work on this hardware since it is specialized. We do have a method to usb boot right into FOS linux for those systems where pxe booting is not an option.

@totoro OK good on the latest version of fog.

Just to make sure I can understand. You can capture an image from this device, but you can’t deploy it because ipxe only sends 26% of the FOS linux image?

Is this device uefi or bios based? If its uefi, what boot loader are you using? ipxe.efi or snponly.efi? If you are using ipxe.efi try the snp.efi or snponly.efi. (the snp.efi will use all interfaces, and snponly will only use the interface it pxe booted from) to chain load linux. The snp driver is built into the nic adapter, where ipxe.efi is much like the linux kernel that has all common network drivers built in. That will lead to the next part.

Another observation is that these device are 2023 models == new hardware. Linux sometimes is a bit slow to take up new hardware. As well as new hardware firmware (uefi/bios) sometimes has bugs in it.

@totoro What version of FOG are you using? I find it strange tat you can capture an image, but then when you restore the image the kernel only transfers 26% and fails.

This sounds like an iPXE issue where it is not sending the FOS image complete to the target computer.

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

Sadly USB boot is not suitable for my use case

You are not getting to the FOG iPXE menu, this is an iPXE issue or as I mentioned before your pxe booting is getting hijacked by that WDS suspicious server. Your firmware doesn’t really indicate what is wrong, it just reboots. So that’s not really helpful in debugging.

You could load ipxe onto a usb stick (yes I know you said that USB wasn’t an option) to see if ipxe will boot from the usb drive for testing. If it does then you know the problem is somewhere along the pxe booting chain and not ipxe itself.

I don’t have the pcap anymore, but looking over the thread, I see this WDS server is responding with a bios boot file (wdsnbp.com) that, if sent to the uefi computer, would cause a uefi computer to download it and just reboot. I wanted to see if the DHCP DISCOVER from the client was saying it was a uefi computer, and then this rouge dhcp server was responding with a bios boot loader. That would be strange but also cause the symptoms.

@sgilbe ok so if we map this out we don’t know then if its a problem with ipxe or the FOS Linux kernel. Since other distros boot it might not be the linux kernel. The only caveat is the other distros have older kernels in the 5.x series.

This next step is to build a fos linux usb boot drive. The basis of the build is this tutorial. https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image

Make sure you read the entire thread so you are aware of the caveats. When you get the boot image built, boot into the Grub menu and select the debug option. The idea is to test booting into FOS linux. If it boots then the issue is with iPXE, if it doesn’t boot then we have a linux issue.

Look at the fog forum chat for a few more hints on building this boot image.

Note to myself: Linux hangs on 4th Gen Xeon Scalable processors

https://www.intel.com/content/www/us/en/products/details/servers/multi-node-server-systems/server-d50dnp.html

I know I’ve seen posts in the forum mentioning booting issues with intel scalable processor line.

@sgilbe Can you live boot a standard linux distro like ubuntu/debian/linux mint?

@sgilbe Set your logging level to 7 in the global FOG configuration. Lets see if that gives a bit more detail to what is going on. With logging level at 1 (default) none of the booting messages are displayed.

Just for reference what hardware are you trying to boot FOS Linux on?

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

the machine I am trying to network boot is a Zotac Zbox

This device may simply not be compatible with iPXE. We can get you setup to usb boot into FOS linux to capture/deploy to this device if iPXE will not work.

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

so you are telling me there is already another instance in the network trying/stealing my network boot

In a word, yes. The problem comes with two different sets of pxe boot instructions sent to your computer, it depends on which offer packet get to the target computer first, is the one it will follow. So like 18 out of 20 pxe boots (made up number to explain a point) dnsmasq may respond first, but the other 2 times this WDS service will respond first making your boot into FOG fail. Its something to look into but as you said, is probably not your problem pxe booting that device.

@gabrielzeit Remember dnsmasq configured this way is also a “dhcp server”

If we look at the packet from the .205 dhcp server, you will see the next server points to .113 (assume to be wds server) and the boot file is surely a wds boot loader (akin to undionly.kpxe). The .1 dhcp server (looks like maybe a router) is also telling the same story.

@gabrielzeit I have to run to a meeting but your pcap is interesting in that you have 3 dhcp servers and 2 of them are pointing at a WDS deployment server.

.201 and .1 are pointing to WDS and .49 appears to be the fog server.

This configuration will cause intermittent issues.

@gabrielzeit said in UEFI: NBP downloaded successfully - then blackscreen:

Then blackscreen. And this repeats every few minutes.

Ensure that secure/safe boot is disabled on the target computer.

@gabrielzeit So just to be clear the device that is hanging with no configuration methods succeeded is a VirtualBox VM and not a physical computer?

If a physical computer is doing this, I would think its spanning tree not something to do with ipxe.

Now if its ipxe then Tom’s recommendation is the better answer. VB is a strange critter as it were. Just be aware that ipxe.pxe is only supported in bios modes and not uefi.

@gabrielzeit Your dnsmasq config is not complete and a bit out of date.

What version of dnsmasq are you running? dnsmasq -v
Here is a tutorial with a good configuration for dnsmasq version 2.75 and later: https://forums.fogproject.org/topic/12796/installing-dnsmasq-on-your-fog-server?_=1693476360006 It will dual boot bios and uefi systems.

@DBCountMan Yes if you are using the location plugin. Using the location plugin you assign the storage node to a location and then when you register the target computer with FOG you assign the target computer to a location. Then when the computer pxe boots it learns who it needs to communicate with. So a storage node would work in this case to spread the load.

One caveat with storage nodes is that they can’t capture images. Only the master node can capture images. Then all storage nodes (including the master node) in the same storage group can deploy the image.

@alexamore90 You have painted an incomplete picture here so I can only speak in general terms.

Using modern target computers you can flood a 1GbE network link with 2-3 simultaneous unicast imaging. Any additional number of simultaneous will impact the overall performance of all systems imaging. On a well designed 1GbE network you should be able to get between 5.5 and 6.5 GB/min as displayed on the partclone screen during imaging. For a 10GbE network you should get between 13 and 15 GB/min. (your performance metrics of 16gbs and 1000 and auto-negotiate are inconsistent). If you only have a 1 GbE network, you can add additional network cards configured in a LAG trunk to spread the imaging load of 10 computers across multiple nic cards.

During imaging the heavy load is carried by the target computer. The FOG server only moves disk blocks from its local storage and sends them out the network adapter. The fog server does monitor the imaging process but does not perform heavy computational load. The speed of the disk subsystem on the FOG server will also have an impact. If your fog server only has 1 physical spinning disk, that disk will have to work hard to keep up with delivering 10 data streams from different locations on the disk platter. In this case if you only have spinning disks, setting up a striped raid configuration using multiple disks will help or moving to SSD drives.

The last comment is if you need to image 10 systems at a time, you might look into FOG multicasting mode. This mode requires some setup and a network infrastructure that supports multicasting, but with this mode you can image 30 or more systems at the same network bandwidth as just 2 multicast streams.

@krokodeilakias said in invalid integer value and chainloading failed:

I’ ll try to strip it down in case I hit on a conflicting command.

This was going to be my next suggestion. Your script is very advanced and I was having a hard time following it just reading through it. As a test discard everything except the first level menu. The error kind of indicates a problem around a colon, but I’m not sure.

@krokodeilakias So lets walk that path then.

You installed FOG 1.5.10 onto your new server, then migrated your database over. So your 1.5.10 version of fog is running on 1.5.7 version of the database? If yes, rerun the fog installer, it will remember all of your previous settings. Half way through the install it will ask you go open a web browser and upgrade the database. That should force the DB to be upgraded to 1.5.10, there has been about 5 years between 1.5.7 and 1.5.10 so there has been database changes and fields added/moved.

@krokodeilakias said in invalid integer value and chainloading failed:

which after successfully loading advanced.php and bg.png returns an error:

OK more clues. This one jumps out at me. In your advanced.php script it references
console --picture https://${fog-ip}/fog/service/ipxe/bg.png

The variable ${fog-ip} doesn’t seem to be set in the advanced.php output. So this might cause an invalid URL. I don’t know if the values fall forward from boot.php or not. If no, then this seems to be an uninitialized variable.

I also noted two invocation lines #!ipxe in the output. That shouldn’t cause a problem because it just resets everything a second time.

@krokodeilakias I don’t have a clue what’s wrong here because your code look right.

If we look at the error message it give me a few clues.

Chainloading failed…

In your script there are only 2 chain commands.

Image deployment works as expected

That would kind of indicate chain -ar https://192.168.61.38/fog/service/ipxe/boot.php##params || chain command is working as intended

but I can’t access Advanced Menu
Indicates that there is a problem with this line: chain -ar https://192.168.61.38/fog/service/ipxe/advanced.php || goto MENU The structure of the line seems OK, I can’t tell at the moment if there are any strange white space characters in there messing us up though.

Lets start out by calling that url directly from your browser, there might be something in the advanced.php program that is having a hard time causing the chain load to fail.