@ALV_SUPECOLES This seems to be similar to this thread: https://forums.fogproject.org/topic/17759/ipxe-initialising-devices
In this case uefi firmware was updated then iPXE stops at initializing devices. If compiling and installing the latest version of iPXE doesn’t solve the problem then we will have to wait for either iPXE developers or Lenovo to fix the problem. https://forums.fogproject.org/topic/15826/updating-compiling-the-latest-version-of-ipxe
@nicolas-moraes We have to be careful some Soho routersthe don’t behave well with pxe booting. Some will put its IP address as the boot server even if you have dhcp option 66 configured. I don’t know the edgerouter, but if there is a bootp option make sure that is turned on too.
The edgerouter is based on vyos so it should behave correctly. Make sure when you define dhcp option 66 its the IP address of the fog server and for dhcp option 67 the boot file name needs to be all lower case. Make sure you don’t have a leading or trailing white space in the name.
If you have double checked everything then get a witness computer and load wireshark on it. Make sure you setup a capture filter of port 67 or port 68 start the scanning then pxe boot the computer until you get the error message then stop scanning.
What you are looking for is the DORA sequence
Discover (sent by target computer, akin to hello world, please configure me)
Offer (sent by one or more dhcp server, I would there to be only one of these and from your edgerouter).
Request (sent by the target computer)
Ack/Nack (sent by the dhcp server)
The interesting packet will be the OFFER.
Look at the packet, there should be an ethernet header section with the following fields.
Next-Server: this should be the IP address of your fog server (this is the bootp part)
Boot-FIle: this should be the name of the ipxe file to load.
If either of those are blank some pxe clients will not boot if they look at the bootp fields.
In the dhcp options section you should see option 66 and 67 that should mirror the values in the bootp part.
@JYost well there is two ways.
I think you will be better served by option 1 in the case of upgrades, but the quickest route is #2 which will download the older binaries for iPXE.
@RocksAndRolls I’m not sure how much additional help I can provide here but where it failing is at the end of the imaging process, the target system logs into the ftp server on the FOG server using the fogproject user account. That fogproject user MUST be able to rename the captured file in /images/dev from the mac address and the move the directory from /images/dev/<mac_address> to /images/<image_name>.
If you have verified you can log into the fog server via ftp as the fogproject user using the password saved in the fog ui then it has to be a permission issue on the /images directory.
The image capture process is
FOS Engine -> connects to /images/dev NFS share and creates a directory that matches it’s mac address
FOS Engine -> Uploads the disk connect as the root user (on the FOS Engine)
FOS Engine -> Connects to the FOG server as fogproject user and issues a mv command to move the directory from /images/dev/<mac_address> to /images/<image_name>
FOS Engine -> Logs out of ftp and completes the cleanup before reboot.
@danieln said in "Windows Other" vs "Windows 10" when creating new Windows 11 images?:
Windows 10 - (9)
This selector was put in place many years ago to deal with the quirks in the OS between the early versions of windows. While there are some minor differences between win10 and win11 disk formats the FOS imaging engine is now robust enough to manage it. So from an imaging standpoint the developers could change the label to “Windows 10&11 - (9)” and be syntactically correct. Who knows what Win 12 will bring.
@rurap said in Slow restoration of Windows 11 with FOG on Proxmox:
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8161] (rev 15)
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) - This is the built-in network card
and the other two are Realtek cards, which appear to be identical.
Technically they are not identical, they have the same family name but one is a 8161 and the other is a 8168 of the rev 15 generation.
Just confirming that the /var/log/messages file exists and the /var/log/syslog one doesn’t ? The 8168 model has been around for years as you can see by the rev numbers. I believe they need nic specific firmware to operate. That should be called out in the boot up messages.
In researching this it seems one instant the 8169 driver was being installed instead of the 8168 linux kernel driver. This will take some looking by lspci -knn | more will list out all installed pci hardware with the “kernel drivers in use” for the hardware. As a hint for looking through the big list the section you are interested in starts with “03:00.0 Ethernet controller” since that is the built in nic you found in the previous post. Lets see if its using the 8169 driver instead.
Edit: I keep seeing reference to these kernel parameters in posts about debugging this nic r8168.aspm=0 r8168.eee_enable=0 pcie_aspm=off Not sure the importance, but noting it here for future reference.
Edit2: This command may help give the answer on driver than looking through the entire list of lspci commands: inxi -Naz
@ITCC Just to be clear you complied the latest version of iPXE using the instructions I provided? If that’s the case and it failed, then there is not much we can do because something in Lenovo’s firmware causing the software to hang. If rolling back the firmware isn’t an option (I don’t know what they were correcting with the update) I think we are stuck unless the iPXE developers have any idea why it might be getting stuck.
@rurap From historical experience if we have an issue with a network card/chip set it will be realtek.
For that realtek nic, can you get the hardware ID of that card, in windows you can get it from the device manager vendor and Id numbers. They will be 4 hex digit codes both the vendor and device id.
You can get them from FOS linux running on the target computer. It will probably be easiest to get from FOS Linux since I will need you to get some info from there for the second part of the question.
Schedule a debug capture/deploy to one of these vostros. Before you hit the scheduled task button tick the debug checkbox. Then schedule the task.
PXE boot the target computer, after several screens of text you need to clear with the enter key you will be dropped to the command prompt.
Key in the following
lspci -nn | grep -i net
Search for the Realtek nic in question and capture the hex codes in the square brackets in the form of [XXXX:XXXX]
Some nics require special firmware to work correctly with linux. Run this command to see if any firmware messages were logged. I should know this by now but I forget if the log is syslog or messages file. So you might have to adjust.
grep -i -e firm /var/log/syslog
Hint: If you want to make it easier for copying and pasting to the target computer do the following after you pxe boot into debug mode.
ip a spasswd make it something simple like hello it will be reset at the next reboot.From there you can copy and paste using the apps.
@ITCC I would start with seeing if snp.efi works instead of ipxe.efi. I have doubts that this will solve the issue, but this is the easiest test.
The second recommendation is to update iPXE to the latest version using this tutorial: https://forums.fogproject.org/topic/15826/updating-compiling-the-latest-version-of-ipxe
The third recommendation is to keep checking for updates to the firmware because it looks like lenovo created an inconsistency in their firmware.
@rurap Well, I thought it said the current version at the top…
OK for a plan B then, with the fog server console, navigate to /var/www/html/fog/service/ipxe directory. In that directory run this command file bzImage. The file command should tell you about that kernel image with a version number embedded in it. Note the version number and post it here.
@rurap said in Slow restoration of Windows 11 with FOG on Proxmox:
I’m not sure what you meant by the FOS kernel
There is a customized linux distro that gets copied to the target computer during pxe booting. You will see that as bzImage and init.xz. To check the version of FOS linux from within the web ui go to fog settings->kernel update. From there it will tell you the version and give you options to update it if needed.
So from your testing you can rule out the OS being deployed. I kind if figured that but ruling it out is always good at helping us narrow down the root of the issue.
So now its down to the network adapter and or the disk controller/nvme. If we could use your idea of process of elimination (one you verify you are on the latest version of FOS Linux) to see where the problem isn’t.
Also we might want to schedule a debug deployment so we can get to the command line on the target system within FOS linux. I would search /var/log/syslog on the target computer for the keyword firmware to see if there are any error messages wanting a specific version of a firmware driver.
@rurap Just to add an additional bit of information the partclone “speed” is a composite rating of
If we had to rule out differences between Dell 7010 and 3910 the decompression of the image is probably not a factor because of the speed differences between a 4th Gen processor and a 12th Gen Intel processor (the Pcore-Ecore might be an issue depending on the FOS kernel). Looking at the truth table I would say the issue is either with #3 or #5.
@rurap OK just to be clear. Your fog server is running on proxmox. You are only restoring to physical computers. The physical computers in question are Dell 7010 (about 10 years old) and Dell Vostro 3910 with a 12 Gen processor writing to a nvme drive.
Windows 10 deploying to both systems are > 6.5GB/min according to partclone
Windows 11 deploying to both systems are < 2GB/min according to partclone
Do I understand the situation correctly?
If this is the case we can probably rule out the fog server and proxmox as a root cause. FOG on a capture or deploy are really not a factor in imaging because at this point its only moving data blocks between the network interface and local nonvolatile storage. All of the heavy lifting is being done by the target computer.
Also, what version of FOG are you running as well as what version of the FOS Linux (fog ui->fog settings->kernel update) are you running. Make sure you are running the latest 6.6.x version. Having an old version of FOG and FOS Linux kernel may cause this type of issue.
@azm9s said in Configuring adding to an active directory domain.:
Do I need to create an capture creation task on the site in the host settings?
Just to be clear on this task. Once you have your golden image the way you want it. Use sysprep command line switches to power off the computer, or user the shutdown -s -t 0 to power off the computer. This will properly close the files for cloning. Then schedule the task for image capture in FOG and then pxe boot the target computer.
Prior to your step file, you need to create a batch script called setupcomplete.cmd in the proper location (google it), and in that batch file use the sc command to enable and start the fog client service (this should be explained in the fog client wiki page). The setupcomplete.cmd batch file (if it exists in the correct location) will be called by OOBE/WinSetup when its all done with its bits. At this point we want the fog client to be enabled and started so the fog client can do its stuff.
Yes on the AD stuff. You will probably want the fog client to rename the computer too, or it will go into AD with a generic system name.
@Mr_____T First let me say I don’t have a clue when it comes to imaging macs. The last time I touched one was in the mid 90s.
But from working on the FOG Forums I know that people are able to image the older ones with FOG. The introduction of the T2 chip caused some issues but I was able to compile a patched version of FOS Linux to allow imaging to work there too.
Looking at your error screen I see several promising clues. iPXE is running and it can see the nic because it can see the mac address of the nic. It doesn’t appear to see the PHY interface because unless you didn’t plug in the network cable, its waiting for the link up to continue.
Did you get iPXE to run by pxe booting or usb boot drive?
I did notice that iPXE is pretty old (circa 2019). You probably want to get the latest version, yes I know you are working with old hardware.
That also raises the question, what version of FOG are you using.
Just to be clear the issue is between iPXE and the network, you haven’t got to FOS Linux bit yet. That may be the next milestone you will need to pass.
@azm9s Yes that needs to be installed before you capture the image, because its needed for post deployment actions (if you want FOG to manage those).
https://docs.fogproject.org/en/latest/installation/client/install-fog-client/
You will install the fog client before capture, but make sure you disable the service before you capture it then use the windows startupcomplete.cmd file that OOBE/WinSetup runs to reenable the service and start it. The issue is if you leave the service enabled, after deployment the fog client (if left running before image capture) will start doing its thing before windows OOBE completes giving you a botched deployment because the fog client program will change the system name and then reboot it, before oobe is ready for the reboot.
I’m not sure what the problem is at the moment (not enough info yet), but this error has nothing to do with the target computer.
“Unable to locate image store” At this point in the deployment, FOS (the customized linux OS that runs on the target system) maps a local directory back to the FOG server. Once it does this then partclone will take over and move the bits from the fog server to the target computer. So where its failing is FOS can’t reach the FOG server or storage node to map the directory.
First thing I would do is look on the fog server console see if in /images/dev there are directory names that look like MAC addresses. If yes then your image uploads are not completing. Then look in /images directory to see if you see directories with names that match the image name you are trying to deploy.
When FOS throws the error unable to locate image store. It will print out a bunch of variables at the end of the error. Those variables are important to debug what went wrong. Take a clear picture of the error with a mobile phone and post it here. Lets match up the variables to what fog thinks its doing.
@azm9s These post install actions are done by the fog agent (client) software that gets installed on the golden image before image capture. Once install and started after windows is done, the FOG agent will rename the computer and then connect it to the domain.
Do you have the fog agent installed in the golden image before image capture?
An alternative to using the fog agent/client is to let the unattend.xml file in windows rename the computer and then connect it to active directory. Either way should work.
@cjiwonder Well this is the strangest pcap that I’ve seen in a while. I finally found a uefi computer pxe booting at 8.3 seconds (you have a very active dhcp network). And the request is from 192.168.200.1.
Your dhcp server is telling the client to get ipxe.efi from 192.168.200.3. Is that your fog server? The dhcp transaction looks normal and from the dhcp side should work.
It looks like you used wireshark on the same subnet as your dhcp server? I would have expected to see broadcast messages from the pxe booting computers instead of unicast messages between the routers and dhcp server. That’s OK because we now know that the dhcp server is sending out the right boot file information (assuming that 200.3 is your fog server). If you would have used the tcpdump command from the fog server we could/should have see the target computer requesting the file to download. That would tell us if the file was actually being sent to the pxe booting computer. But from the dhcp side it looks good.
@Paladin1 First the wipe speed is controlled by the wipe method + target hard drive + processor speed of the target computer. The FOG server and/or network has no impact on the speed it takes to wipe a hard drive.
You really can’t do much about improving the wipe speed with the hardware. The hardware you have is all you have. The wipe method is a sliding scale between efficiency and security. A one pass data wipe will be of course faster than 3 to 7 pass DoD mandated wipe (be aware this DoD wipe is only really effective for HDD, SSDs are good with just a single pass wipe.
So using the FOG utility to fast wipe a hard drive uses a single pass wipe method. The full wipe method uses a 3 pass wipe (i.e. writes to every block on the drive 3 times).
You can use an alternative disk wiping tool like dban and have fog send that tool to the target computer via PXE booting if the standard FOG wipe command is not what you are looking for.
Lastly you need to be aware of technical limitations of the media you have. For spinning disk I’ve typically seen on average for a single disk a maximum speed of 90MB/s (I have seen as high as 120MB/s for certain drives) If you were going to use a single pass wipe on a 2TB hard drive it would take 7 hours-7 minutes-11 seconds to write to every block on that drive.
For SATA SSD drives I’ve seen on average 500MB/s for that same 2TB ssd drive it would take 1 hour-16 minutes-53 seconds
And to round out the numbers for NVMe drives on average write speeds of 1500MB/s it would take about 26 minutes for a single pass wipe