Posts made by george1421

@COE Typically if the permissions on the /images directory get messed up you can reinstall FOG and it will fix the permissions.

If you run the chmod 777 /images that will make the directory world writable but not the existing files / folders under that directory. If you use chmod -R 777 /images it will change everything under that directory to world writable.

But before you do that, the linux user fogproject should have read write access to /images. The password for that user is found in the hidden file /opt/fog/.fogsettings That is the user account fog uses to move files around while imaging.

@Bristow-0 Is this a physical computer or VM? Is it possible to add a new storage device to the FOG server?

Something to understand that the fog database that lists the images is disconnected from the raw data file. You can move the image files to some place else, it won’t impact the FOG UI until you try to deploy a previously captured image.

@CheekiBreekiHelsinki you have some missing information needed in your first post.

1. What device is your dhcp server? (manufacturer and version)
2. Is it on the same subnet as the FOG server? You mentioned the VM and FOG server was on the same subnet.
3. What hypervisor are you using?
4. I assume when you say “run PXE” you are saying you are trying to pxe boot the VM?
5. Is the VM in bios or uefi mode?
6. What specifically do you have defined for dhcp option 66 and 67?

Lets start with those questions to see what’s next

@nec Good find, but I would also say that you are missing the boot-file (or whatever its called) at the pool level. That is the bootp part of the pxe booting. When you look at the pcap from the witness computer on the same subnet as the pxe booting computer. You should see both sets of values set.

I have no idea what Kea dhcp server is but the config file looks similar to the linux standard ISC-DHCP dhcp server.

@nec There is two sets if fields that need to be updated. There are the bootp fields in the ethernet header next-server and boot-file AND the dhcp fields option 66 and 67 that need to be set. Depending on who programmed the pxe boot code the target computer could look at either set of values (bootp or dhcp). Many dhcp servers set both just to be sure.

You should use wireshark on a witness computer with the capture filter of port 67 or port 68 Start wireshark and pxe boot the target computer to the error. Stop collecting data and look at the OFFER packet you should have one packet for every dhcp server that hears the DISCOVER packet.

Make sure if you only expect one dhcp server to respond that there is only one offer packet. Now look at the offer packet, in the ethernet header the next-server and boot-file is properly specified. Also validate the dhcp values 66 and 67. My bet there is something missing in the offer packet. If you can’t find the issue upload the pcap here and we can look at it.

@jmeyer Make sure you are paying attention to MB/s and Mb/s. It goes without saying there is a difference.

I wrote an article about 8 years ago now. https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast

This contained some benchmarks I did at the time. FOG imaging speed is (according to Partclone) is made up of several elements. FOG Server disk subsystem speed, the speed at which the fog server and move the image from the disk to network interface, network transfer time, the client receiving the file and expanding it in memory, and finally the client moving the data to local storage. All of those go into the number displayed by partclone.

For clarity let me present some theoretical best network speeds.

For a 100Mb/s network link the maximum transfer speed is 12.5MB/s or 750MB/m
For a 1GbE network link the maximum transfer speed is 125MB/s or or 7.5GB/m
For a 10GbE network link the maximum transfer speed is 1250MB/s

Pay attention if your network speed is getting capped at or around one of the maximum transfer speeds. I’ve seen someone in the past only get 8MB/s transfer rate. He/she had 1GbE on each end, but between the ends there was a network switch link that was running in 100Mb/s half duplex. Not saying that is your case, but stuff happens.

Now why I referenced that article above. It has the commands to benchmark your hardware. iperf3 will give you network speeds between the network interfaces and the kernel. It has nothing to do with moving fog image blocks between systems. If you put the FOS target system in debug mode you can run iperf3 between FOS Linux and the FOG server. This will give you an idea of the bandwidth you have. I would do this with a computer that is exhibiting the slow imaging speed and then one that has normal imaging speed. Lets see if the network speeds are compatible. I’m not willing to rule out is the linux kernel version 6.1.x vs 6.6.x it maybe there is something that is missing in FOS linux for this new hardware.

When you have the slow target computer in debug mode run this command to see if the kernel is complaining about the hardware/firmware. grep -i -e "firm" /var/log/syslog That should return any lines that complain about needing special firmware for the hardware.

@jmeyer said in what USB can support iPXE boot:

Do you mean using brand adapteur certified for PXE such as :

Exactly. They have to say pxe boot and then be branded to the device manufacturer. UEFI firmware is not like linux (well it is minux/linux) but in a general purpose linux it has one of every common driver, where uefi only needs drivers for the hardware actually installed in the device, there is no need for a 3com isa driver in a Dell laptop with a 14 gen processor (contrasting something really old with something new).

Typically if the uefi bios sees a uefi compatible nic adapter in the computer, the F12/boot manager will list pxe booting as an option. If its not listed then the network device is not supported by the uefi firmware.

@jmeyer said in Error PXE-E18 - Lenovo ThinkPad E16:

It’s now loading but terribly slow (average 6 Mb/s on a Gb link).

What is loading slow iPXE or the image via partclone? I haven’t been following closely the issue but I think I saw that someone tried an earlier version of the FOS kernel, like in the 6.1 range and imaging on these current lenovos went at normal speeds. I don’t know what the linux developers did between 6.1 and 6.6 to cause this slowness. Have we identified what nic adapter is installed in these computers? We would need the hardware ID of the nic to research it.

@robertkwild Yes right now pxe booting issue is between uefi firmware and network adapter. Typically the hardware vendor will have a recommended usb adapter (typically hardware vendor branded $$) that supports pxe booting, where the driver for the usb adapter is built into uefi.

I the imaging process once bzImage is loaded (FOS kernel) and you have a network issue then the issue with with the “FOG kernel”. You are not there yet in the booting process.

@jmeyer said in Error PXE-E18 - Lenovo ThinkPad E16:

I’ll put a wireshark tomorow to look what happen.

You really need to see what the client is being told and by who to try to explain why bios works and uefi doesn’t. There is something unknown going on here.

@robertkwild You have two things at play here.

1. The laptops don’t have built in nics so you can pxe boot them.
2. If you don’t purchase usb nic drivers that is supported by the laptop vendor they will not pxe boot.

You have to consider that uefi bios is much like a linux kernel. The uefi firmware vendor has control over what usb nics they support for pxe booting. If the laptop vendor doesn’t support the nic you will not be able to pxe boot. This isn’t a fog issue, its between the laptop vendor and the nic vendor.

The second issue is when ipxe boots it must have the correct nic driver onboard to boot.

The third issue bzImage boots that is the FOS Linux kernel. That is in the realm of fog developers. If the driver is available for linux, it can be included in bzImage. You will have better luck if the usb nic is a bit older and a bit more generic and linux will probably have a driver.

Now if you have a usb nic that is supported by linux but not the vendor, you can always usb boot right into FOS linux (bzImage) and bypass pxe booting altogether. This won’t work in a campus environment but will work on the imaging bench. I think I have a tutorial to usb flash drive into iPXE to get more normal imaging experience. Booting right into bzImage has a few caveats.

@davidsontiago If you are using sysprep, include the command line option to tell sysprep to power off the computer. You have this message because when you pick shutdown from the menu, windows doesn’t actually shutdown the computer but put it into a sleep state leaving some files open that FOG sees. Sysprep will power off the computer correctly for cloning or use the following command “shutdown -s -t 0” to properly power off the source computer and close all open files.

@jmeyer ok you did a lot of the initial debugging so its not the easy stuff.

The bit harder stuff is using either tcpdump on the fog server if its on the same subnet as the pxe booting computer. Or a witness computer loaded with wireshark on the target computers subnet.

If the fog server and pxe booting computer are on the same subnet you can use this tutorial to capture a pcap file: https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue you can look at the pcap with wireshark or post it here and I will look at it.

or if your target computer is on a different subnet than the fog server you will need to use a witness computer with wireshark loaded. Use a capture filter of “port 67 or port 68” to only grab the DORA (dhcp) packets on the network.

What you want to look for is there will be a discover packet from the target computer that says “hello I’m here, come configure me”.

You will get a OFFER from one or more dhcp servers. This is the packet you are interested in. In the OFFER there will be a header section with the fields next-server and boot-file. These need to be populated with the fog servers IP address and snponly.efi (or whatever). This is the bootp fields. In addition if you look down in the dhcp options 66 and 67 that should match what is in next-server and boot-file fields. I’m suspecting something is not right with this section.

This is a pxe boot (iPXE) issue and unrelated to what FOS kernel or version of FOG you are running.

@DBCountMan Its been several years since I last messed with it, but if I remember correctly we needed the nsfv4 components compiled into FOS OS. I have the tools to do that for the testing, but I thought those changes were merged into the inits, but I’m not sure without looking. They can and should be there since there is zero impact to nfs v3.

You are right its a lot of I tried this and it worked instead of a clear tutorial. If I can make some time I should try to write up a more formal how to.

@DBCountMan Its not officially supported but in the feature request I put in so many years ago I had it working.

@lperoma Or you can leave fast startup enabled and either have sysprep power off the computer or use the shutdown command to properly close the file system for imaging. I think the shutdown command is shutdown.exe -s -t 0 to properly closed all files.

@kbats183 I haven’t looked at the quick registration bit of code but I did have a hack that updated the fog.auto.reg (full registration) option to begin imaging right after registration without a reboot. This involved taking the FOG supplied script updating it to your requirements and then adding a bit of code onto the end and then finally dynamically patching that script on every boot of the target system. It sounds like a lot of steps and its complicated but not if you know how fog works.

I should be able to point you in a direction so you can get started.

When FOG Linux (the os that gets transferred to the target computer for imaging) boots it runs the master fog script stored in the /bin directory in FOS Linux. The scripts bits of that /bin directory is here: https://github.com/FOGProject/fos/tree/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin The file you are interested in is called fog.auto.reg. If you can understand bash script programming you can modify this file to fit your requirements. Once you have the script the way you need it then you can dynamically patch when FOS linux boots using this tutorial (understand this tutorial is for patching fog.man.reg but the concenpt is the same only the file name changes.

https://forums.fogproject.org/topic/13500/dynamically-patching-fos-using-a-postinit-script

I have tutorials that are targeting the manual registration but the concepts are similar. (hint: this is where I tweak the script to give me a custom calculated host name kind of like the auto naming but better. )
https://forums.fogproject.org/topic/14278/creating-custom-hostname-default-for-fog-man-reg

So that’s the background, how do I make it image right after rebooting?
For the fog.man.reg file you append a bit of code that makes the script think the system rebooted and collects the latest imaging info.
(For the life of me I can’t find that tutorial I created but here is the script that I collected from a recent issue with the code in the script.

    sysuuid=$(dmidecode -s system-uuid)
    sysuuid=${sysuuid,,}switch
    mac=$(getMACAddresses)
    base64mac=$(echo $mac | base64)
    token=$(curl -Lks --data "mac=$base64mac" "${web}status/hostgetkey.php")
    curl -Lks -o /tmp/hinfo.txt --data "sysuuid=${sysuuid}&mac=$mac&hosttoken=${token}" "${web}service/hostinfo.php" -A ''
    curl -Lks -o /tmp/hinfo.txt --data "sysuuid=${sysuuid}&mac=$mac" "${web}service/hostinfo.php" -A ''
    [[ -f /tmp/hinfo.txt ]] && . /tmp/hinfo.txt
    . /bin/fog.download 

In the case of fog.man.reg this code would be added at the very end of the script.

Looking at the fog.auto.reg script I would say it should go at the very end of that script too.

ref: https://forums.fogproject.org/topic/17601/deploy-image-right-after-registration-without-a-reboot/3
ref: https://forums.fogproject.org/topic/16378/start-imaging-right-after-the-full-host-registration-without-reboot-possible

@Bhav In addition to what Tom posted, the server response timeout is suspicious too. IF the target computer is in uefi mode and it downloaded undionly.kpxe the target computer should have responded with something about undionly.kpxe is not the right format. But in this case its getting a server response timeout. This makes me think two things. 1) your dhcp opition 66 is set incorrectly or the name of the file has a capitalization issue. For linux Undionly.kpxe and undionly.kpxe are not the same thing. Either way I would start with your dhcp server and make sure dhcp options 66 and 67 are set correctly for a uefi based computer.

@Quintin-Giesbrecht said in FOG Very Slow to Deploy Image - Lenovo Neo 50Q Gen 4:

Lenovo Neo 50Q Gen 4

I’m with Tom here, I would go with the latest FOS Linux kernel with this new hardware. You didn’t happen to mention what version of the FOS Linux kernel you are using (FOG UI -> FOG Configuration -> Kernel Update). I would go with the latest 6.6.x or later kernel. This is the easiest and quickest way to see if it solves the problem. It would be useful to know what version is your current kernel too so if we see this issue again we can help the next guy.

@lperoma Thank you for the pcaps. There is something strange going on here.

This is what I gleaned from our discussion (actors)

192.168.0.40 is dnsmasq
192.168.3.253 is my gateway (and DHCP server).
192.168.0.21 Fog Server
192.168.0.0/255.255.252.0 /23 network

When I look at the pcap the target computer 192.168.3.8 is downloading undionly.kpxe from the dnsmasq server 0.40 and not the fog server 0.21. At no time in the pcap do I see 0.21 in the dialog. Also it looks like the target computer is on a different subnet than the dhcp server and fog server. The pcap only shows one side of the conversation. I can see the Discover and Request bits of DORA. I’m missing what the dhcp server(s) are telling the target computer. This is because the client is on the other side of some router. The target computer is indeed a bios base system (HP) and the 0.40 server is running as a VM under proxmox.

So there are several questions here.

1. Is your fog server really 192.168.0.21
2. Why does the dnsmasq server have FOG’s pxe boot files? Those should be on the fog server.
3. Why does dnsmasq only tell the client computer to boot from itself and not the fog server?
4. You are capable of causing a good pxe boot and not good pxe boot. What are you doing different, or what is different between the two pcaps?

It would be interesting to see the other side (what the client is being told) by taking a third computer loaded with wireshark to the clients subnet and then use a capture filter of port 67 or port 68 to see what actors are involved and what the client is being told. At this time I’m only interested in what is not working.

ref: