FOG does not copy NVRAM/EFI Bootorder

[lucjayjay]

Hi, I just set up FOG 1.5.7 yesterday and noticed a big disadvantage compared to our current image solution with clonezilla.
FOG’s Partclone does not clone the NVRAM content, which results in machines not knowing how to boot or booting in the wrong order because EFI is not informed about the Partition/Bootloader changes.

Is this the desired behavior? Because this completely kills the advantage of Fog to remote imaga pc’s unattended.

FYI, this is the tool Clonezilla uses to clone the NVRAM:
https://github.com/stevenshiau/clonezilla/blob/master/sbin/update-efi-nvram-boot-entry

[Sebastian Roth]

@lucjayjay Thanks for your post and the information. We are aware of this topic but we don’t seem to get many requests on this by users. I don’t have a huge set of UEFI hardware to test but I suspect that most UEFI firmware implementations do some “magic” and find the Windows bootloader themselves. See my post here: https://forums.fogproject.org/topic/13922/dell-optiplex-aio-5270

You are very welcome to help us add the NVRAM backup feature to FOG. I suppose a first step would be to add this through post deploy scripting for testing. The FOG developer and testing team is sparse at the moment (has been for months) and we have not found the time to get into that topic.

[lucjayjay]

@Sebastian-Roth All our Client Workstations run with UEFI so for us this is a really important topic. We probably could edit the bootorder after imaging ourselves with the postinstall script and efibootmgr that is included in FOS.

But I dont know how I can differentiate the PC models/Images in my Script and I would have to edit that Script for every new Image if the bootorder for that image/system is different.

So my preferred way would be to replace partclone with clonezilla that supports copying the NVRAM natively. But without knowing why Partclone is used (pros/cons over clonezilla) and how the whole Image Task works that seems quite hard to do by myself, given that Partclone is a core part of the whole FOG architecture.

[Sebastian Roth]

@lucjayjay said in FOG does not copy NVRAM/EFI Bootorder:

All our Client Workstations run with UEFI so for us this is a really important topic.

Many people use FOG on UEFI machines now and so far the “mass of complains” about this being an issue has not happened. Don’t get me wrong. I am not saying this is not an issue at all or we don’t care about it. Just saying that we didn’t have many requests about this over the last months eventhough I hear many people use FOG with UEFI machines already.

But I dont know how I can differentiate the PC models/Images in my Script and I would have to edit that Script for every new Image if the bootorder for that image/system is different.

Differentiating between images shouldn’t be much of a problem. Though as I said I see this as a first step of testing. Not saying this is gonna be the solution in the end.

So my preferred way would be to replace partclone with clonezilla that supports copying the NVRAM natively.

As you can read on their website Clonezilla is “Based on Partclone (default), Partimage (optional), ntfsclone (optional), or dd to image or clone a partition”. So it’s not because we use partclone but just that we haven’t implemented backing up NVRAM information yet as Clonezilla has.

[lucjayjay]

@Sebastian-Roth

I can tell you that running this command in FOS

efibootmgr -c -d /dev/sda -p 2 -L “IXPE Hardisk” -l “\EFI\ipxe\bootx64.efi”

solves the problem for me but this isnt a universal approach since disk,partition and Filepath could be different for every scenario.

I have read it is possible to feed partclone custom scripts while imaging but couldn’t find any information on this online. Maybe u have some insight on this and could tell me where in the FOG source code I would have to look for the partclone command and how to insert custom commands.

[Sebastian Roth]

@lucjayjay As mentioned you should be able to use FOG’s post deploy scripts. Search the forum on this term. Let us know if you need more detailed description on this.

[lucjayjay]

@Sebastian-Roth

thats what i did, i put that command in the post deploy script. But as i said this isn’t a universal solution. ATM im looking at the Clonezilla implementation but i would need to know were in the FOG Sourcecode the partclone command is build and how i could hook it.

[Quazz]

@lucjayjay You don’t need to worry about partclone, this would just be ran afterwards (since all it will do is try and look for an EFI bootable file and write those to the NVRAM if the entry doesn’t exist yet, which means it has to run after partclone is fully finished)

I’m not sure if you can just grab that script as is and try and call it like that with options, though. It seems like it tries to grab some stuff from other files too, not entirely sure what’s in those.

This will likely require some more investigation for implementation, but you can try it already with post deploy scripts as said.

[Sebastian Roth]

@lucjayjay As we said earlier, this part has nothing to do with partclone. If you want to start looking on where to put this in FOG/FOS (the FOG Linux OS) you’d probably start here: https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.download#L269

thats what i did, i put that command in the post deploy script. But as i said this isn’t a universal solution.

Would you mind sharing your current script here? I am fairly sure we can give you some advances hints on how to make it more universal. Again, this is not about trying to prevent this from being added to FOG itself but it’s always worth to try and refine things before we add it.

[lucjayjay]

@Quazz Clonezilla creates a efi-nvram.dat from the cloned disk when creating an image. This file contains the output of

efibootmgr -v

and is used to recreate the correct labels for the bootmenu.
My idea would be to maybe implement the https://gitlab.com/stevenshiau/clonezilla/blob/master/sbin/update-efi-nvram-boot-entry logic into FOG, because rewriting that from scratch would be too much work for me atleast.
I will try and see if I can integrate it into the post deploy script but without the efi-nvram.dat it is not complete.

I dont know if integrating Clonezilla Sourcecode into FOG would be legal tho since Clonezilla uses a GPL License and FOG uses MIT.

[Tom Elliott]

@lucjayjay where do you see fog uses an MIT license? We use gpl version 3. The client might have MIT though, and from legal standpoint there would be little issue

[Sebastian Roth]

@lucjayjay @Tom-Elliott Seems like the FOG community scripts are published under the MIT license. All the rest should be GPL 3 I think.

I think reusing the script you found is a great idea. Give it a go. We’ll happily support you.

[lucjayjay]

Quick Update, im having a look at the Clonezilla Source but it seems they are only creating one boot entry https://gitlab.com/stevenshiau/clonezilla/blob/v3.37.14/sbin/update-efi-nvram-boot-entry#L630

This is not suitable for my environment because we have 2 different EFI Files on the EFI Partition that need to be added. First is ipxe.efi that points to fog and the other being the windows bootloader. Btw do u see any problem with this approach instead of configuring every machine to network boot ?

I honestly don’t understand why they are creating a copy of the nvram but don’t use it to recreate the exact same boot entrys on the target machine but instead are manually checking the efi partition for efi files (https://gitlab.com/stevenshiau/clonezilla/blob/v3.37.14/sbin/update-efi-nvram-boot-entry#L228) and only use the one that’s found first.

Maybe im overseeing something but I think recreating the NVRAM from the efi-nvram.dat file would be much easier especially when ur cloning to machines of the same type.

Leading to my next question, is there any way to run custom scripts when cloning (to create a nvram copy) ? And how do I distinguish which Image is being deployed to my machine in the post deploy script ?

@Sebastian-Roth I also found that the UEFI firmware implementations does some “magic” and finds the Windows bootloader themselve, like u said

Thanks in advance.

[Sebastian Roth]

@lucjayjay said in FOG does not copy NVRAM/EFI Bootorder:

First is ipxe.efi that points to fog and the other being the windows bootloader. Btw do u see any problem with this approach instead of configuring every machine to network boot ?

Good point! I have to admit that I have not tested this myself yet and I don’t know anyone who has. Possibly there are some advanced users here in the forums who have done this?! Most people just use standard UEFI PXE boot I think. The FOG iPXE menu chainloads your machines back to disk by default in case there is no task scheduled for it.

Maybe im overseeing something but I think recreating the NVRAM from the efi-nvram.dat file would be much easier especially when ur cloning to machines of the same type.

This sounds like a straight forward solution. Though I am not sure is it might cause a lot of trouble if you deploy to a (slightly) different machine. We need testing and it’s really great you are pushing this forward!!

Leading to my next question, is there any way to run custom scripts when cloning (to create a nvram copy) ? And how do I distinguish which Image is being deployed to my machine in the post deploy script ?

You should be able to use the so called post init scripts for now. We will work on adding this to the official scripts as soon as we have a nice solution.

[Greg Plamondon]

I know this isn’t a very good fix but It works for me…
Windows boot manager always gets added to the top of the list in boot order, I found this powershell script here:

This PowerShell script moves the first non windows entry to the top of the list.

# This script looks for the first non-Windows Boot Manager entry in the UEFI/GPT boot order and moves it to the top
# For preventing newly installed Windows from hijacking the top boot order spot on my UEFI/GPT image testing VMs
# by mmseng
# https://github.com/mmseng/bcdedit-revert-uefi-gpt-boot-order

# Notes:
# - There's very little point in using this on regular production machines being deployed. Its main use is for machines being repeatedly imaged, or might be useful for lab machines.
# - AFAICT bcdedit provideds no way to pull the friendly names of the devices in the overall UEFI boot order list. Therefore, this script only moves the first entry it identifies in the list which is NOT "{bootmgr}" (a.k.a. "Windows Boot Manager"). It's up to the user to make sure the boot order will exist in a state where the desired result is achieved.
# - In my case, my test UEFI VMs initially have the boot order of 1) "EFI Network", 2) whatever else. When Windows is installed with GPT partitioning, it changes the boot order to 1) "Windows Boot Manager", 2) "EFI Network", 3) whatever else. In that state, this script can be used to change the boot order to 1) "EFI Network", 2) "Windows Boot Manager", 3) whatever else.
# - This functionality relies on the completely undocumented feature of bcdedit to modify the "{fwbootmgr}" GPT entry, which contains the overall list of UEFI boot devices.
# - AFAICT bcdedit is really only designed to edit Windows' own "{bootmgr}" entry which represents one of the "boot devices" in the overall UEFI list.
# - Here are some sources:
#   - https://www.cnet.com/forums/discussions/bugged-bcdedit-349276/
#   - https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/bcd-system-store-settings-for-uefi
#   - https://www.boyans.net/DownloadVisualBCD.html
#   - https://serverfault.com/questions/813695/how-do-i-stop-windows-10-install-from-modifying-bios-boot-settings
#   - https://serverfault.com/questions/714337/changing-uefi-boot-order-from-windows


# Read current boot order
echo "Reading current boot order..."
$bcdOutput = cmd /c bcdedit /enum "{fwbootmgr}"
echo $bcdOutput

# Kill as many of the stupid characters as possible
echo "Removing extraneous characters from boot order output..."
$bcdOutput = $bcdOutput -replace '\s+',''
$bcdOutput = $bcdOutput -replace '`t',''
$bcdOutput = $bcdOutput -replace '`n',''
$bcdOutput = $bcdOutput -replace '`r',''
$bcdOutput = $bcdOutput.trim()
$bcdOutput = $bcdOutput.trimEnd()
$bcdOutput = $bcdOutput.trimStart()
$bcdOutput = $bcdOutput -replace ' ',''
echo $bcdOutput

# Define a reliable regex to capture the UUIDs of non-Windows Boot Manager devices in the boot order list
# This is difficult because apparently Powershell interprets regex is a fairly non-standard way (.NET regex flavor)
# https://docs.microsoft.com/en-us/dotnet/standard/base-types/regular-expressions
# Even then, .NET regex testers I used didn't match the behavior of what I got out of various Powershell commands that accept regex strings
# However this seems to work, even though I can't replicate the results in any regex testers
$regex = [regex]'^{([\-a-z0-9]+)+}'
echo "Defined regex as: $regex"

# Save matches
echo "Save strings matching regex..."
$foundMatches = $bcdOutput -match $regex

# Grab first match
# If Windows Boot Manager (a.k.a. "{bootmgr}" was the first in the list, this should be the second
# Which means it was probably the first before Windows hijacked the first spot
# Which means it was probably my "EFI Network" boot device
$secondBootEntry = $foundMatches[0]
echo "First match: $secondBootEntry"

# Move it to the first spot
echo "Running this command:"
echo "cmd /c bcdedit $bcdParams /set `"{fwbootmgr}`" displayorder $secondBootEntry /addfirst"
cmd /c bcdedit $bcdParams /set "{fwbootmgr}" displayorder $secondBootEntry /addfirst```

[JJ Fullmer]

@Greg-Plamondon another suggestion. We had enough issues with pxe boot compatibility acorss different devices that we embed a custom bootloader that has an option for booting directly to a local copy of the ipxe.efi file. We use grub2win and just have its installation as part of our provisioning scripts that are started by sysprep firstlogon. Found this to work better than trying to get custom nvram type settings, this way you get it individuallized per hardware but also using a universal config file.