Posts made by george1421

@skyborne98 Its still a password issue. Look in the storage node configuration document for the storage node that is accepting this image. There is a ftp user ID and password in that configuration document, my guess is the password defined there is not in line with the actual linux account.

@herman Debugging this may be difficult. What changed between 5GB/min and now 270MB/min?

I’m going to suspect network issues, but what specifically changed?

@davidgoncalves If you look in /images/dev directory do you see sub directories there that appear to be mac addresses? These should be consistent with mac address of the source computer. Under normal conditions there should not be any of these directories in the /images/dev folder unless there is an active capture going on.

@alexamore90 Let me make sure I understand the problem here.

You ave two esxi servers one is Dell and one is HP. The Dell has 3 VMs that are FOG servers 1, 2, 3. The HP has only one FOG server A (as an example).

So your problem is that on the Dell when you have multiple unicast deployments you have on average 3GB/m deployment rate, but on the HP you only get 1/2GB/m rate. Is that your problem?

If that is the problem, the first thing I don’t know is what is the differences (hardware wise) between the Dell and HP (CPU cores, RAM, network bandwidth) Also how busy is the Dell vs HP servers? Are these ESXi boxes performing any other tasks?

I doubt its the FOG server that is the root of the problem here. As a test you can export FOG1 as an OVA file and then import it onto the HP. Stop FOG1 on the Dell and then power it up on the HP. Do you see the performance differences still between the Dell and HP? If yes than the problem is with the HP and not specifically with FOG.

@alexamore90 My intuition is giving me doubt in a few things here.

First lets get our scale correct so that we can be sure we are talking about the same things.

The speed you mentioned is probably from partclone image deployment. If yes then that scale time frame is volume per minute. So when you image 10 pcs at a time you get 3GB/minute and a single pc you can get 16-17GB/m (I have doubt on this single pc speed)

Let me explain, you have a 1GbE network switch, so your link from the ESXi box to the tp switch is 1GbE. So 1 GbE == 1000 bits per second == 125MB/s == 7.5GB/min (theoretical speed) In theory its possible to get to 16GB/m but its doubtful.

The number you see in the partclone screen is a composite speed. That is the combined speed for the fog server to move a block of data from its local storage, the time it takes to transmit the data over the network, network transport time, then on the client to receive the block of data, decompress the data and write it to the target computer’s local storage media.

So since that speed in partclone is a composite score, its possible to have a partclone speed faster than the theoretical speed of a 1GbE network connection. If you have a very fast target computer and a highly compressed image.

Now with a single 1GbE network link in testing I’ve saturated that network link at 3 unicast imaging. Past 3 simultaneous unicast imaging with a saturated link performance drops off quite a bit.

On a well managed 1GbE network I would expect to see 6-6.5GB/m transfer rates. On a 10GbE core network with 1GbE access layer I would expect to see between 13 and 15 GB/m.

Almost all of the heavy load of imaging is done by the target computer. The fog server only moves data from its local storage, to the network adapter and then monitors the overall imaging process. I can run FOG on a raspberry pi server and get 5GB/m transfer rates. The size of the fog server really doesn’t have an impact on imaging speed as long as the server can get a data block from its local storage media and out the network adapter.

So how can you make FOG imaging go fast if you want/need to have multiple unicast imaging streams going?

1. Have a fast disk array with multiple spinning disks or SSD disk.
2. Have either multiple ethernet adapters configured in a LAG configuration to your network switch or run a 10GbE link between your fog server and network infrastructure.
3. Use enterprise class network switches. Those TP switch are low end switches with limited backplane speed. Even used / older enterprise (managed) switches may be faster.

If you can’t change any of the above, and you need to image multiple computers at the same time consider looking into FOG multicast imaging.

@sgilbe Well my first attempt to rebuild the kernel gave me the same results as you. Not what I expected so I need to work a bit more. If I can get something that boots in the next day or so, are you willing to test to see if it resolves your booting issue?

@jjurak Is this something you are actively looking to do?

If yes, do you use the fog client program? If yes that will be the problem. The fog client is hard coded to use port 80. I’ve been asking for a few years to decouple the fog client communication port from the management ui. But there has been other issues that have taken priority.

If you are interested in doing this I can probably give you a few areas to look.

@sgilbe Do you still have access to this server?

@Numa09 First this looks like a DOS Batch file not a VBS file. That is maybe why its giving you an error with line 2.

Second make sure that csv file is part of your snapin pack.

@Sebastian-Roth I did do a side by side comparison between ubuntu configs and FOS linux configs and there are roughly 1800 differences. Many were in drivers and options. The only one that stood out in the efi section was CONFIG_EFI_MIXED which allowed a 32 bit EFI kernel boot a 64 bit linux kernel. Seems kind of strange, but we probably should turn that on.

Though a second process I started with an ia64 defconfig template and then added in the FOS required settings leaving almost all of the defconfig settings in place but adding in the fog required settings. I built this last night but haven’t had time to see if it boots. I did not add in the old ISA card network drivers or network adapters that I’m pretty sure are not in circulation like DEC Tulip network drivers. That kernel came in at 15MB as compared to the FOS kernel of 10MB. I’m not really worried about that extra 5MB kernel in size in 2023. This kernel is based on linux 6.5.3.

The other thing I need to point out is the the OPs platform is a server with an intel scalable processor. I don’t know what other hardware might be getting in the way. The FOS kernel should at least try to boot, it might not boot completely but should at least try to boot. We are not seeing that. By building the FOS usb boot drive we have eliminated all of the pxe and ipxe issues so we’ve narrowed it down to the FOS kernel, and swapping in the ubuntu kernel points directly to the FOS kernel at fault.

I hadn’t considered a ubuntu kernel patch to be the solution here either. I used linux 6.5.3 thinking that it should have all of the mainstream patches already in it.

@sgilbe said in Client hangs at EFI stub::

This kernel is hanging at the same place as the fos kernel

In a way that’s good because it should be the same exact kernel except for 6.5.3 version instead of 6.2.x. So now the next part is I need to compare ubuntu kernel settings with FOS linux kernel settings.

These should be the steps needed to configure network/pxe booting using pfsense firewall: https://forums.fogproject.org/topic/17021/how-to-configure-pfsense-for-netbooting

If you use a pfSense router as your primary dhcp server you can configure it to provide dynamic PXE booting information to your target computers that you wish to image.

To configure PXE (Network) Booting with pfSense please do the following.

1. Edit your DHCP Server configuration for the subnet where you require pxe booting.
2. At the top of the DHCP Server conifugration page for the network interface (LAN in this example). Make sure that you have the DHCP server enabled and Ignore Bootp queries disabled.
   
3. Scroll down the page until you see the Network Booting section and press the Display Advanced button.
   
4. Tick the checkbox for Enable network booting and then enter the IP address of the FOG server (netboot server) in the Next Server field.
5. Complete the form as below
   
6. Make sure you don’t have any value populated for the TFTP server field
   
7. As well as under the Additional Bootp/DHCP options section. Make sure you are not populating dhcp options 60, 66, or 67.
   
8. Save the configuration and then restart the dhcp server.
   9, Be aware that you might need to enable some additional firewall rules if you have the LAN inteface locked down. Net/PXE booting requires access to the TFTP server on udp port 69 on the FOG server.

Note: If you are running pfSense as a VM, you may need to configure promiscuous mode on the network interface for your hypervisor. You would only need this option if your external target computer’s can’t pick up an IP address from pfSense. This promiscuous mode is a requirement of dhcp and not network booting.

@FuriousGamer065 Is the lab on its own subnet or can it be placed on its own subnet (still fully routable across your campus)? The goal would be to put this lab on its own broadcast domain. Then as you said use dnsmasq to configure it as a proxydhcp server to supply pxe boot info only. Run dnsmasq on the FOG server that way when the fog server is off line there is no pxe boot info and everything functions normally on this lab subnet.

@sgilbe It looks like my build environment was really out of date. The last time I needed to create a one off kernel was for version 5.15.x. More to the point I updated the build environment to 6.5.3 and built this kernel. https://drive.google.com/file/d/1P-OX1LXhm-N_oBLg0PVcIjj0P3Cxm_Rp/view?usp=drive_link download this kernel and save it onto your flash drive as bzImage. I don’t expect it to work any better than the FOS standard kernel, but I want to see if the new kernel release works on your processor. If this kernel doesn’t work better than the stock FOS kernel then I compare what ubuntu is creating with this config file to see what is missing. I do have kernel options to turn on the advanced features of the Intel scalable processor, but the base x64 kernel should also run on this processor (IMO).

Also please confirm that you updated all of the firmware on the server using the lifecycle controller.

@sgilbe I realize this debugging is a lot of try this and do that, but we’ve narrowed it down to exactly the FOS Linux kernel. The ubuntu kernel error bombed out exactly where I expected it to do, at mounting the virtual hard drive (init.xz).

I think you asked earlier about the config file fog uses these are all posted on the fog project github site: https://github.com/FOGProject/fos/tree/master/configs The config file you are looking for is kernelx64.config this is the config file used to create the current kernel.

@Redbob ok I think I know where its falling down but we need evidence and not an opinion.

Can you get a computer on the target’s network running wireshark? If yes create a pcap of the pxe booting process with this capture filter port 67 or port 68 This will only capture the dhcp process. What we are interested in is the dhcp OFFERS from one or more dhcp servers. In one of those OFFER packets is the bad actor that is sending your pxe booting into a loop. If you can’t figure it out who the bad actor is post the pcap here and I’ll look at it. The answer will be in the pcap. Use my capture filter so we only see the dhcp process and not other random traffic on your network.

@Redbob Yes again we are seeing the multiple queries and then to load request. You have confirmed that 172.24.5.180 is the fog server?
Did you turn on any firewalls on the fog server, or are they on?

For a test, on a windows computer on the same subnet as the pxe booting computer, install the tftp client feature. Temp turn off the windows firewall (or this test will fail). Finally from a windows command line key in the following command tftp 172.24.5.180 get undiohly.kpxe see if you can download the file to your test computer on the remote subnet.

@sgilbe I haven’t had a chance to look into the settings as of now. Its kind of a complicated subject to explain in an email. My gut reaction is that it should boot even if the advanced features are not turned on. It might be what rhromkyo mentioned about the preboot environment but that also got me thinking if this is new hardware have you run through the lifecycle controllers and updated all of the firmware in that server? I’m going to keep working on the kernel bits but lets make sure there isn’t something in the firmware causing the issue.

Also as a hack and it won’t fully work, it would be interesting to take the vmlinuz from your ubuntu disk that is working and save it on the usb stick as bzImage (watch the case) and see if the kernel tries to boot when you select debug mode from the grub menu. It appears that the FOS linux kernel doesn’t even try to boot.

@Redbob I think something happened to the pcap where it didn’t get uploaded correctly. I get file not found when I go to download it