Posts made by george1421

@jmeyer Make sure you are paying attention to MB/s and Mb/s. It goes without saying there is a difference.

I wrote an article about 8 years ago now. https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast

This contained some benchmarks I did at the time. FOG imaging speed is (according to Partclone) is made up of several elements. FOG Server disk subsystem speed, the speed at which the fog server and move the image from the disk to network interface, network transfer time, the client receiving the file and expanding it in memory, and finally the client moving the data to local storage. All of those go into the number displayed by partclone.

For clarity let me present some theoretical best network speeds.

For a 100Mb/s network link the maximum transfer speed is 12.5MB/s or 750MB/m
For a 1GbE network link the maximum transfer speed is 125MB/s or or 7.5GB/m
For a 10GbE network link the maximum transfer speed is 1250MB/s

Pay attention if your network speed is getting capped at or around one of the maximum transfer speeds. I’ve seen someone in the past only get 8MB/s transfer rate. He/she had 1GbE on each end, but between the ends there was a network switch link that was running in 100Mb/s half duplex. Not saying that is your case, but stuff happens.

Now why I referenced that article above. It has the commands to benchmark your hardware. iperf3 will give you network speeds between the network interfaces and the kernel. It has nothing to do with moving fog image blocks between systems. If you put the FOS target system in debug mode you can run iperf3 between FOS Linux and the FOG server. This will give you an idea of the bandwidth you have. I would do this with a computer that is exhibiting the slow imaging speed and then one that has normal imaging speed. Lets see if the network speeds are compatible. I’m not willing to rule out is the linux kernel version 6.1.x vs 6.6.x it maybe there is something that is missing in FOS linux for this new hardware.

When you have the slow target computer in debug mode run this command to see if the kernel is complaining about the hardware/firmware. grep -i -e "firm" /var/log/syslog That should return any lines that complain about needing special firmware for the hardware.

@jmeyer said in what USB can support iPXE boot:

Do you mean using brand adapteur certified for PXE such as :

Exactly. They have to say pxe boot and then be branded to the device manufacturer. UEFI firmware is not like linux (well it is minux/linux) but in a general purpose linux it has one of every common driver, where uefi only needs drivers for the hardware actually installed in the device, there is no need for a 3com isa driver in a Dell laptop with a 14 gen processor (contrasting something really old with something new).

Typically if the uefi bios sees a uefi compatible nic adapter in the computer, the F12/boot manager will list pxe booting as an option. If its not listed then the network device is not supported by the uefi firmware.

@jmeyer said in Error PXE-E18 - Lenovo ThinkPad E16:

It’s now loading but terribly slow (average 6 Mb/s on a Gb link).

What is loading slow iPXE or the image via partclone? I haven’t been following closely the issue but I think I saw that someone tried an earlier version of the FOS kernel, like in the 6.1 range and imaging on these current lenovos went at normal speeds. I don’t know what the linux developers did between 6.1 and 6.6 to cause this slowness. Have we identified what nic adapter is installed in these computers? We would need the hardware ID of the nic to research it.

@robertkwild Yes right now pxe booting issue is between uefi firmware and network adapter. Typically the hardware vendor will have a recommended usb adapter (typically hardware vendor branded $$) that supports pxe booting, where the driver for the usb adapter is built into uefi.

I the imaging process once bzImage is loaded (FOS kernel) and you have a network issue then the issue with with the “FOG kernel”. You are not there yet in the booting process.

@jmeyer said in Error PXE-E18 - Lenovo ThinkPad E16:

I’ll put a wireshark tomorow to look what happen.

You really need to see what the client is being told and by who to try to explain why bios works and uefi doesn’t. There is something unknown going on here.

@robertkwild You have two things at play here.

1. The laptops don’t have built in nics so you can pxe boot them.
2. If you don’t purchase usb nic drivers that is supported by the laptop vendor they will not pxe boot.

You have to consider that uefi bios is much like a linux kernel. The uefi firmware vendor has control over what usb nics they support for pxe booting. If the laptop vendor doesn’t support the nic you will not be able to pxe boot. This isn’t a fog issue, its between the laptop vendor and the nic vendor.

The second issue is when ipxe boots it must have the correct nic driver onboard to boot.

The third issue bzImage boots that is the FOS Linux kernel. That is in the realm of fog developers. If the driver is available for linux, it can be included in bzImage. You will have better luck if the usb nic is a bit older and a bit more generic and linux will probably have a driver.

Now if you have a usb nic that is supported by linux but not the vendor, you can always usb boot right into FOS linux (bzImage) and bypass pxe booting altogether. This won’t work in a campus environment but will work on the imaging bench. I think I have a tutorial to usb flash drive into iPXE to get more normal imaging experience. Booting right into bzImage has a few caveats.

@davidsontiago If you are using sysprep, include the command line option to tell sysprep to power off the computer. You have this message because when you pick shutdown from the menu, windows doesn’t actually shutdown the computer but put it into a sleep state leaving some files open that FOG sees. Sysprep will power off the computer correctly for cloning or use the following command “shutdown -s -t 0” to properly power off the source computer and close all open files.

@jmeyer ok you did a lot of the initial debugging so its not the easy stuff.

The bit harder stuff is using either tcpdump on the fog server if its on the same subnet as the pxe booting computer. Or a witness computer loaded with wireshark on the target computers subnet.

If the fog server and pxe booting computer are on the same subnet you can use this tutorial to capture a pcap file: https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue you can look at the pcap with wireshark or post it here and I will look at it.

or if your target computer is on a different subnet than the fog server you will need to use a witness computer with wireshark loaded. Use a capture filter of “port 67 or port 68” to only grab the DORA (dhcp) packets on the network.

What you want to look for is there will be a discover packet from the target computer that says “hello I’m here, come configure me”.

You will get a OFFER from one or more dhcp servers. This is the packet you are interested in. In the OFFER there will be a header section with the fields next-server and boot-file. These need to be populated with the fog servers IP address and snponly.efi (or whatever). This is the bootp fields. In addition if you look down in the dhcp options 66 and 67 that should match what is in next-server and boot-file fields. I’m suspecting something is not right with this section.

This is a pxe boot (iPXE) issue and unrelated to what FOS kernel or version of FOG you are running.

@DBCountMan Its been several years since I last messed with it, but if I remember correctly we needed the nsfv4 components compiled into FOS OS. I have the tools to do that for the testing, but I thought those changes were merged into the inits, but I’m not sure without looking. They can and should be there since there is zero impact to nfs v3.

You are right its a lot of I tried this and it worked instead of a clear tutorial. If I can make some time I should try to write up a more formal how to.

@DBCountMan Its not officially supported but in the feature request I put in so many years ago I had it working.

@lperoma Or you can leave fast startup enabled and either have sysprep power off the computer or use the shutdown command to properly close the file system for imaging. I think the shutdown command is shutdown.exe -s -t 0 to properly closed all files.

@kbats183 I haven’t looked at the quick registration bit of code but I did have a hack that updated the fog.auto.reg (full registration) option to begin imaging right after registration without a reboot. This involved taking the FOG supplied script updating it to your requirements and then adding a bit of code onto the end and then finally dynamically patching that script on every boot of the target system. It sounds like a lot of steps and its complicated but not if you know how fog works.

I should be able to point you in a direction so you can get started.

When FOG Linux (the os that gets transferred to the target computer for imaging) boots it runs the master fog script stored in the /bin directory in FOS Linux. The scripts bits of that /bin directory is here: https://github.com/FOGProject/fos/tree/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin The file you are interested in is called fog.auto.reg. If you can understand bash script programming you can modify this file to fit your requirements. Once you have the script the way you need it then you can dynamically patch when FOS linux boots using this tutorial (understand this tutorial is for patching fog.man.reg but the concenpt is the same only the file name changes.

https://forums.fogproject.org/topic/13500/dynamically-patching-fos-using-a-postinit-script

I have tutorials that are targeting the manual registration but the concepts are similar. (hint: this is where I tweak the script to give me a custom calculated host name kind of like the auto naming but better. )
https://forums.fogproject.org/topic/14278/creating-custom-hostname-default-for-fog-man-reg

So that’s the background, how do I make it image right after rebooting?
For the fog.man.reg file you append a bit of code that makes the script think the system rebooted and collects the latest imaging info.
(For the life of me I can’t find that tutorial I created but here is the script that I collected from a recent issue with the code in the script.

    sysuuid=$(dmidecode -s system-uuid)
    sysuuid=${sysuuid,,}switch
    mac=$(getMACAddresses)
    base64mac=$(echo $mac | base64)
    token=$(curl -Lks --data "mac=$base64mac" "${web}status/hostgetkey.php")
    curl -Lks -o /tmp/hinfo.txt --data "sysuuid=${sysuuid}&mac=$mac&hosttoken=${token}" "${web}service/hostinfo.php" -A ''
    curl -Lks -o /tmp/hinfo.txt --data "sysuuid=${sysuuid}&mac=$mac" "${web}service/hostinfo.php" -A ''
    [[ -f /tmp/hinfo.txt ]] && . /tmp/hinfo.txt
    . /bin/fog.download 

In the case of fog.man.reg this code would be added at the very end of the script.

Looking at the fog.auto.reg script I would say it should go at the very end of that script too.

ref: https://forums.fogproject.org/topic/17601/deploy-image-right-after-registration-without-a-reboot/3
ref: https://forums.fogproject.org/topic/16378/start-imaging-right-after-the-full-host-registration-without-reboot-possible

@Bhav In addition to what Tom posted, the server response timeout is suspicious too. IF the target computer is in uefi mode and it downloaded undionly.kpxe the target computer should have responded with something about undionly.kpxe is not the right format. But in this case its getting a server response timeout. This makes me think two things. 1) your dhcp opition 66 is set incorrectly or the name of the file has a capitalization issue. For linux Undionly.kpxe and undionly.kpxe are not the same thing. Either way I would start with your dhcp server and make sure dhcp options 66 and 67 are set correctly for a uefi based computer.

@Quintin-Giesbrecht said in FOG Very Slow to Deploy Image - Lenovo Neo 50Q Gen 4:

Lenovo Neo 50Q Gen 4

I’m with Tom here, I would go with the latest FOS Linux kernel with this new hardware. You didn’t happen to mention what version of the FOS Linux kernel you are using (FOG UI -> FOG Configuration -> Kernel Update). I would go with the latest 6.6.x or later kernel. This is the easiest and quickest way to see if it solves the problem. It would be useful to know what version is your current kernel too so if we see this issue again we can help the next guy.

@lperoma Thank you for the pcaps. There is something strange going on here.

This is what I gleaned from our discussion (actors)

192.168.0.40 is dnsmasq
192.168.3.253 is my gateway (and DHCP server).
192.168.0.21 Fog Server
192.168.0.0/255.255.252.0 /23 network

When I look at the pcap the target computer 192.168.3.8 is downloading undionly.kpxe from the dnsmasq server 0.40 and not the fog server 0.21. At no time in the pcap do I see 0.21 in the dialog. Also it looks like the target computer is on a different subnet than the dhcp server and fog server. The pcap only shows one side of the conversation. I can see the Discover and Request bits of DORA. I’m missing what the dhcp server(s) are telling the target computer. This is because the client is on the other side of some router. The target computer is indeed a bios base system (HP) and the 0.40 server is running as a VM under proxmox.

So there are several questions here.

1. Is your fog server really 192.168.0.21
2. Why does the dnsmasq server have FOG’s pxe boot files? Those should be on the fog server.
3. Why does dnsmasq only tell the client computer to boot from itself and not the fog server?
4. You are capable of causing a good pxe boot and not good pxe boot. What are you doing different, or what is different between the two pcaps?

It would be interesting to see the other side (what the client is being told) by taking a third computer loaded with wireshark to the clients subnet and then use a capture filter of port 67 or port 68 to see what actors are involved and what the client is being told. At this time I’m only interested in what is not working.

ref:

@lperoma TBH I never look at the DORA process from the server side because it only tells us half the story. It looks like its working right in both flows.

Are you sure the dnsmasq is configured on the .40 server as a dhcp proxy or is it doing traditional dhcp functions. In a dhcp proxy config it only sends the boot file override command and then at the end of the DORA process it will then answer the dhcp proxy request on port 4011. Your dnsmasq configuration should look similar to the one in this tutorial: https://forums.fogproject.org/topic/12796/installing-dnsmasq-on-your-fog-server

When I try to debug this process I use the process outlined in this document: https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

Since your dnsmasq server isn’t on your FOG server, I would run the tcpdump command from your dnsmasq server since there is some unicast communications to dnsmasq you would miss if you ran it on the fog server. It would be intersting to see what the client is being told on the pxe booting process that fails.

Minus the dhcp proxy communications you would miss you could also use wireshark with the capture filter of port 67 or port 68 running on a witness computer on the same IP subnet as the pxe booting target computer. There may be an unknown dhcp server mucking up the works on this pxe booting.

If you are willing to share the pcap I will look at it to see if there is anything that jumps out at me as the problem.

@LiamRetrams I can’t give you the exact steps to solve this but I can point you in a direction.

You will need to solve this in the windows realm. Once oobe starts FOG imaging is out of the picture so we can’t do anything in a post install script. You will need to use the bcdedit commands from inside windows to reset the boot order. You should be able to do this within the startupcomplete.cmd batch file or using the unattend.xml file in the first run section, or possibly deploy a fog snapin to run a batch file with the bcdedit commands.

I would manually reset the boot order to the way you want it then from within windows use the bcdedit commands to look at how it is ordered via bcdedit. That should give you the structure of what you need to do with the bcdedit commands to rebuild that format on a different system.

@lperoma said in Chainloading failed, hit 's' for the iPXE shell:

192.168.3.253

What device is that (manufacturer and model)? is that a soho router?

How does dnsmasq fit into this design? Is it setup as a proxy dhcp server?

The reason why I ask is I’ve seen many soho routers say (tell client computers) they are the boot server even if you have dhcp option 66 set correctly. The bios on the target computer is getting ipxe loaded correctly. Once iPXE starts it issues a dhcp discovery request again to find the boot server’s IP address. What it is getting this time is your gateway’s address.

With this new information I don’t think its spanning tree.

@lperoma So I can make the generalization statement that time fixes your issue?

What I want you to test is this.

PXE boot into the error and then press s to get to the ipxe console.

Wait 20 seconds.

Now key in the following at the console prompt
chain tftp://192.168.3.253/default.ipxe
Does this work and continue into the FOG iPXE menu. In that the only thing different between not working and working is time? (I’m driving to a point here).

If time fixes the issue, lets place a dumb switch (unmanaged cheap-o like monoprice) switch between the building network switch and the target computer. Now pxe boot the computer does it boot correctly? If yes then its possibly a spanning tree issue with your building network switch. Make sure your building switch has fastSTP, portfast or whatever your switch mfg calls the fast spanning tree protocol. I might be off base here but it sounds like it might be spanning tree. I know that processor is pretty old, but I don’t think its cpu related.

@lperoma Undionly.kpxe and snp.efi are probably the safest choices. Both of these use the built in driver built into the network adapter. The ipxe.pxe and ipxe.efi are more like the linux kernel where it has every known driver built into the boot loader. All of the other ipxe boot loaders are for unique situations that are not common.

So your system is an old 4th gen i5 processor. That’s strange that ipxe is misidentifying that CPU.