UEFI Boot Install of XenServer Close but no Cigar

dangbird

Hello All - Was wondering if someone can get me over the hill and get the Cigar lit.
Happy to share any files needed.
We use XenServer as a Hypervisor and deploy many servers each week. Bios mode works AMAZING with Fog.
Here is the Menu Entry we use where we call a Script with an Answer file to load over the network:

kernel http://${fog-ip}/iso/xen82/boot/xen dom0_max_vcpus=4 dom0_mem=1024M,max:1024M com1=115200,8n1 console=com1,vga
module http://${fog-ip}/iso/xen82/boot/vmlinuz xencons=hvc console=hvc0 console=tty0 answerfile=http://${fog-ip}/iso/xen82/scripts/answerfile-82 install
module http://${fog-ip}/iso/xen82/install.img
boot

I based the above off of the post here:
https://ipxe.org/appnote/xenserver

The new hardware we are receiving has NVMe drives and will only work with UEFI Loaded XenServer. From what I can tell the XenServer Uses grub64.efi as per this post under “Configuring your Server for TFTP Boot UEFI” https://docs.citrix.com/en-us/xenserver/7-1/install/network-boot.html

That said I have the DHCP Setup and working great, and Fog boots right into UEFI Menu, but I cannot figure out the correct Grub/Install file to use in the correct format due to my pure ignorance.

So, Anyone that has a suggestion I can try I would appreciate it much!

george1421

@dangbird Ok got it to boot in uefi mode.

I didn’t play around any and test different options once I decided to ‘follow’ the instructions.

This is just phase 1 but I can get it to boot. Phase 2 is making it a single menu to boot both bios and uefi. But lets make sure we can get uefi running on your campus.

This is what I have for my FOG iPXE menu parameter

chain tftp://${fog-ip}/EFI/xenserver/grubx64.efi

Now the path MUST BE EXACT as I defined in the /tftpboot directory. You can play with getting http working on your own dime.

So in the tftpboot directory create the /EFI/xenserver and move these files into there.

grub.cfg grubx64.efi install.img vmlinuz xen.gz

You will have to create grub.cfg from the web site you lined. I just pasted this into that file

menuentry "XenServer Install (serial)" {
 multiboot2 /EFI/xenserver/xen.gz dom0_mem=1024M,max:1024M watchdog dom0_max_vcpus=4 com1=115200,8n1 console=com1,vga
 module2 /EFI/xenserver/vmlinuz console=hvc0
 module2 /EFI/xenserver/install.img
}

pxe booted into the FOG iPXE menu and selected Xen installer menu.

When it loads the VM went dark for about 8 seconds then started booting vmlinuz. It was a long enough pause that I went “crap it didn’t work again”, but then the linux kernel started up.

george1421

@dangbird That above boot parameters seem a bit nonsensical because they are loading two kernels into memory one with kernel and one with module. But only the kernel loaded module will start with the boot command.

I’ve found that most current linux kernels support both bios and uefi. So it should work for linux.

For the uefi boot. Since it uses grub to start things off, see if you can locate the grub.cfg file and post that here. Lets see what they are doing different for uefi mode.

dangbird

@george1421

Sorry about the delay, email notification did not hit me.
the Nonsensical entry is a clear display of my ignorance on the boot site of linux.

I found TWO grub configs, both under the EFI folder.
Here is “grub.cfg” - I will Post the Second Grub as an additional reply for clarity.

set default="0"

function load_video {
  insmod efi_gop
  insmod efi_uga
  insmod video_bochs
  insmod video_cirrus
  insmod all_video
}

load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2

set timeout=5

menuentry "install" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
    module2 /boot/vmlinuz xencons=hvc console=hvc0 console=tty0
    module2 /install.img
}

menuentry "no-serial" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M console=vga
    module2 /boot/vmlinuz console=tty0
    module2 /install.img
}

menuentry "safe" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M nosmp noreboot noirqbalance no-mce no-bootscrub no-numa no-hap no-mmcfg iommu=off max_cstate=0 nmi=ignore allow_unsafe com1=115200,8n1 console=com1,vga vga=keep
    module2 /boot/vmlinuz xencons=hvc console=hvc0 console=tty0
    module2 /install.img
}

menuentry "multipath" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
    module2 /boot/vmlinuz xencons=hvc console=hvc0 console=tty0 device_mapper_multipath=enabled
    module2 /install.img
}

menuentry "shell" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
    module2 /boot/vmlinuz xencons=hvc console=hvc0 console=tty0 bash-shell
    module2 /install.img
}

dangbird

@dangbird

Here is the USB version “grub-usb.cfg”

set default="0"

function load_video {
  insmod efi_gop
  insmod efi_uga
  insmod video_bochs
  insmod video_cirrus
  insmod all_video
}

load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2

set timeout=5
search --file --set /install.img

menuentry "install" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
    module2 /boot/vmlinuz xencons=hvc console=hvc0 console=tty0
    module2 /install.img
}

menuentry "no-serial" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M console=vga
    module2 /boot/vmlinuz console=tty0
    module2 /install.img
}

menuentry "safe" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M nosmp noreboot noirqbalance no-mce no-bootscrub no-numa no-hap no-mmcfg iommu=off max_cstate=0 nmi=ignore allow_unsafe com1=115200,8n1 console=com1,vga vga=keep
    module2 /boot/vmlinuz xencons=hvc console=hvc0 console=tty0
    module2 /install.img
}

menuentry "multipath" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
    module2 /boot/vmlinuz xencons=hvc console=hvc0 console=tty0 device_mapper_multipath=enabled
    module2 /install.img
}

menuentry "shell" {
    multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
    module2 /boot/vmlinuz xencons=hvc console=hvc0 console=tty0 bash-shell
    module2 /install.img
}

george1421

@dangbird well this one has me a bit stumped. But lets try this out and see where it goes.

Edit: Ugh, I just reread the OP and you already had this bit worked out

Copy the 3 files zen.gz, vmlinuz and install.img into /tftpboot/xen on the fog server.

using this grub menu item:

menuentry “install” {
multiboot2 /boot/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
module2 /boot/vmlinuz xencons=hvc console=hvc0 console=tty0
module2 /install.img
}

Lets translate that into iPXE format with this:

kernel tftp://${fog-ip}/xen/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
module tftp://${fog-ip}/xen/vmlinuz xencons=hvc console=hvc0 console=tty0
module tftp://${fog-ip}/xen/install.img
boot || goto menu

So lets create a new iPXE menu with this structure

Menu Item: os.xen.install
Description: XEN Installer
Parameters:
kernel tftp://${fog-ip}/xen/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
module tftp://${fog-ip}/xen/vmlinuz xencons=hvc console=hvc0 console=tty0
module tftp://${fog-ip}/xen/install.img
boot || goto MENU
Menu Show with: All Hosts

Will it work ?? Testing will tell us.

george1421

@dangbird ok looking at this in a new light. When you set the dhcp option 67 to ipxe.efi do you get to the iPXE menu? From there, when you pick the xen server install what do you get for an error message?

dangbird

@george1421
Yes everything boots…I get the Fog Menu in EFI mode and I have the DHCP set such that it detects Bios-Legacy or EFI boot.
I can use Fog to EFI Load machines all day long.
As you indicated I can boot this XenServer and Load it via FOG in Legacy mode PERFECT, and its awesome using the http protocol of the version you suggested.

However if I switch to EFI boot, and use the same menu, I get IPXE Exec format error.
“http://fogiphere/boot/xen.gz …Ok”
“Could not select: Exec format error (http://ipxe.org/2e008081)”
See Pic.
![0_1607364734731_ipxe.jpg](Uploading 0%)

george1421

@dangbird Ok a couple of things you are using the http protocol so you will need to place the files in /var/www/html/iso directory. In your OP I see you are calling xen, but in your last post I see xen.gz. The error that iPXE is throwing is that the file doesn’t look bootable.

Can you make those files available so I can try to spin them up in our testlab?

dangbird

@george1421

YES!
Here you go.
Let me know what else you need.
https://drive.google.com/drive/folders/1UqKsUVnkOIlsva54WOhpDAu4-1OTABe0

george1421

@dangbird Ok I can pxe boot it using my menu in bios mode, but in uefi mode it fails. I did this on a ESXi VM. The only thing that changed between fail and working is UEFI -> BIOS change.

If under linux I run file xen I get this

file xen
xen: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, BuildID[sha1]=562b9323885fffea752a2e4f1308833a8fbcf8e0, stripped

for vmlinuz I get

vmlinuz: Linux kernel x86 boot executable bzImage, version 4.19.0+1 (mockbuild@a50e0282e69b) #1 SMP Fri Apr 12 12:44:47 UT, RO-rootFS, swap_dev 0x6, Normal VGA

Notable that these are both 32 bit kernels and I’m suspecting not UEFI compatible. iPXE rejects both xen and vmlinuz in uefi mode. Basically saying it doesn’t know the file format.

Do you have an installer image that is UEFI format?

george1421

@dangbird I’m thinking that we need grub to do the magic here. I have a few ideas, let me work with it for a bit. I need to go and grab a few more files from the file share.

dangbird

@george1421
Yes if you just burn the files I provided to USB, and install on a system that is EFI / UEFI only it will pull
/EFI/xenserver/grubx64.efi then /boot/vmlinuz then /install.img

I THINK the magic file is grubx65.efi OR /boot/efiboot.img that it pulls…

So by the system being in UEFI boot Only mode it will boot and load from that same USB…
In otherwords the files I provided burned to USB with rufus etc will boot EFI USB…

According to Citrix Xenserver people they insist the TFTP boot magic for UEFI boot is Here:
https://support.citrix.com/article/CTX217680 But I cannot get it to go with Fog…
I set my DHCP to provide grubx64.efi and can get a Grub menu from FOG but think I am barking up the wrong tree.

I placed a USB Image file ready to UEFI Boot in the same Shared Google Folder as before to try to make it easier.
“UEFIUSBINSTALLER.IMG”
I appreciate the effort.

george1421

@dangbird said in UEFI Boot Install of XenServer Close but no Cigar:

I set my DHCP to provide grubx64.efi and can get a Grub menu from FOG but think I am barking up the wrong tree.

Actually this is the tree I’m climbing at the moment.

No problem on the burn of files. I don’t need them post working/giving up.

george1421

@dangbird Ok got it to boot in uefi mode.

I didn’t play around any and test different options once I decided to ‘follow’ the instructions.

This is just phase 1 but I can get it to boot. Phase 2 is making it a single menu to boot both bios and uefi. But lets make sure we can get uefi running on your campus.

This is what I have for my FOG iPXE menu parameter

chain tftp://${fog-ip}/EFI/xenserver/grubx64.efi

Now the path MUST BE EXACT as I defined in the /tftpboot directory. You can play with getting http working on your own dime.

So in the tftpboot directory create the /EFI/xenserver and move these files into there.

grub.cfg grubx64.efi install.img vmlinuz xen.gz

You will have to create grub.cfg from the web site you lined. I just pasted this into that file

menuentry "XenServer Install (serial)" {
 multiboot2 /EFI/xenserver/xen.gz dom0_mem=1024M,max:1024M watchdog dom0_max_vcpus=4 com1=115200,8n1 console=com1,vga
 module2 /EFI/xenserver/vmlinuz console=hvc0
 module2 /EFI/xenserver/install.img
}

pxe booted into the FOG iPXE menu and selected Xen installer menu.

When it loads the VM went dark for about 8 seconds then started booting vmlinuz. It was a long enough pause that I went “crap it didn’t work again”, but then the linux kernel started up.

george1421

@george1421 and for the bonus round bringing the two configurations together to give you one menu for both bios and uefi booting.

Note: I did not test this only glued the code together. YMMV

iseq ${platform} efi goto os.xen.uefi

kernel tftp://${fog-ip}/EFI/xenserver/xen.gz dom0_max_vcpus=1-16 dom0_mem=max:8192M com1=115200,8n1 console=com1,vga
module tftp://${fog-ip}/EFI/xenserver/vmlinuz xencons=hvc console=hvc0 console=tty0
module tftp://${fog-ip}/EFI/xenserver/install.img
boot || goto MENU

:os.xen.uefi
chain tftp://${fog-ip}/EFI/xenserver/grubx64.efi || goto MENU

dangbird

@george1421
WOW - Thats great. I am getting the Boot to Fog Menu, then Selecting the new item with the chain, it starts the boot flashes REAL Fast “syntax error” then kicks to grub> prompt. So I am sure I missed a step…I even dos2unix 'ed the grub.cfg file.
Anyhow, I am guessing its all on my end now. I appreciate it very much. IF I get this working I will do a write up on what I am doing to help others. I did start with trying to get the http working but swapped to direct TFTP and followed your instructions. So I will grab another beverage and start again to see if I can make your magic work.

george1421

@dangbird When I created the grub.cfg file I used putty to connect to the FOG server and used vi and pasted the text directly in from copying it from the web page. I didn’t change anything. Just watch your case on the file path, because case is important.

As for grub you may not have luck switching to http if grub wasn’t compiled with http support. tftp may be your only option. Just saying.

Now also be aware, while I was playing I also placed a grub.cfg in a directory called /grub because I was testing a number of paths. Being thrown into the grub shell means that it can’t find the grub.cfg file. But if you are getting a syntax error that means it must be finding the grub.cfg script.

Lastly all of my testing was done on an ESXi VM. I did not try on real hardware.

dangbird

@george1421
FOR SURE your way works PERFECTLY.
I forgot I cannot use $ inside grub.cfg for any variables…
SO you got it working for me.
I APPRECIATE you very much. Let me know any specifics of how I can contribute in return.
I will work on a “Load XenServer from Scratch using FOG” write up at a minimum.

PROBLEM SOLVED after I took out my variables and used your Copy and Paste VI etc.

THANK YOU!

george1421

@dangbird said in UEFI Boot Install of XenServer Close but no Cigar:

Let me know any specifics of how I can contribute in return.
I will work on a “Load XenServer from Scratch using FOG” write up at a minimum.

This is perfect payment in kind. Thank you

UEFI Boot Install of XenServer Close but no Cigar

244

11.6k

17.1k

154.6k