Deployment and Quick Delete Host not working from iPXE menu

Richardpofnz

Hi All,

Have a strange issue which I can’t quite figure out.

This is a clean install of Fog on 1.5.6 booting to ipxe.efi.

I can register hosts from the iPXE menu and any deployment/capture tasks configured from the web GUI are working.

Now when I attempt to either use Deployment or Quick Delete Host from the iPXE menu it prompts for a username and password. I enter the default “fog” and “password” which appears to fail but without any prompts. Note, these credentials still work in the GUI. After three attempts it bypasses the menu and I end up here:

Any guidance would be greatly appreciated.

Cheers

Sebastian Roth

@Richardpofnz Sorry for the very long delay to respond. Just couldn’t find the time to test this up till now.

When I do “Deploy Image” or “Quick Host Deletion” from the menu I also get the login menu but if I type the wrong credentials I get a message saying “Invalid login!” and get back to the main iPXE menu. I can do this over and over again and it will not bypass the login menu as far as I have tried.

The picture you posted looks exactly the way it would when you select to boot “Client System Information (Compatibility)” from the iPXE menu. Any chance you got things mixed up here a bit?

If this is still the case, could you possibly the a short video of the whole thing?

Richardpofnz

This post is deleted!

Richardpofnz

@Sebastian-Roth

Hi Sebastian,

No worries regarding the delayed reply.

I’m definitely selecting Deploy Image and Quick Host Deletion. The only difference is when I select “Deploy Image” it prompts three times for the username and password and only once for “Quick Host Deletion” before proceeding to the client system information page. The only modification I’ve made to the iPXE menu is removing the memtest option.

Video available here:
https://drive.google.com/open?id=1dQGGnlj5cVPVvwoXWKmqy7kEEoSFa0sb

Excuse the time it takes to go through the NIC selection (there are 3 in this workstation).

Thanks
Richard

Sebastian Roth

@Richardpofnz Thanks for the video. Watched it a couple of times but I still have no clue what’s wrong. Guess we need to take a look at the menu code FOG returns to the client. Please open the following URL in your web browser: http://192.168.x.y/fog/service/ipxe/boot.php?mac=aa:bb:cc:dd:ee:ff (put in the correct FOG server IP and client MAC address)

Copy and paste the whole text here to the forums. Don’t worry about disguising the network information as it’s all internal IP addresses anyway. Not much anyone in public can really use to attack you.

Richardpofnz

@Sebastian-Roth said in Deployment and Quick Delete Host not working from iPXE menu:

http://192.168.x.y/fog/service/ipxe/boot.php?mac=aa:bb:cc:dd:ee:ff

Hi Sebastian,

I was just being over the top with the internal IP masking. Happy to leave it unmasked to make things easier.

Output below:

#!ipxe
set fog-ip 192.168.25.16
set fog-webroot fog
set boot-url http://${fog-ip}/${fog-webroot}
cpuid --ext 29 && set arch x86_64 || set arch i386
goto get_console
:console_set
colour --rgb 0x00567a 1 ||
colour --rgb 0x00567a 2 ||
colour --rgb 0x00567a 4 ||
cpair --foreground 7 --background 2 2 ||
goto MENU
:alt_console
cpair --background 0 1 ||
cpair --background 1 2 ||
goto MENU
:get_console
console --picture http://192.168.25.16/fog/service/ipxe/bg.png --left 100 --right 80 && goto console_set || goto alt_console
:MENU
menu
colour --rgb 0x00567a 0 ||
cpair --foreground 1 1 ||
cpair --foreground 0 3 ||
cpair --foreground 4 4 ||
item --gap Host is registered as MWAT-001!
item --gap -- -------------------------------------
item fog.local Boot from hard disk
item
item fog.keyreg Update Product Key
item fog.deployimage Deploy Image
item fog.multijoin Join Multicast Session
item fog.quickdel Quick Host Deletion
item fog.sysinfo Client System Information (Compatibility)
choose --default fog.local --timeout 30000 target && goto ${target}
:fog.local
chain -ar ${boot-url}/service/ipxe/grub.exe --config-file="find --set-root /BOOTMGR;chainloader /BOOTMGR"" || goto MENU
:
kernel memdisk initrd=memtest.bin iso raw
initrd memtest.bin
boot || goto MENU
:fog.keyreg
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param keyreg 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.deployimage
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param qihost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.multijoin
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param sessionJoin 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.quickdel
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param delhost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.sysinfo
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=127000 web=http://192.168.25.16/fog/ consoleblank=0 rootfstype=ext4 storage=192.168.25.16:/images/ storageip=192.168.25.16 loglevel=4 mode=sysinfo
imgfetch init_32.xz
boot || goto MENU
:bootme
chain -ar http://192.168.25.16/fog/service/ipxe/boot.php##params ||
goto MENU
autoboot

Sebastian Roth

@Richardpofnz What jumped at me is this part:

...
item fog.local Boot from hard disk
item
item fog.keyreg Update Product Key
...
:fog.local
chain -ar ${boot-url}/service/ipxe/grub.exe --config-file="find --set-root /BOOTMGR;chainloader /BOOTMGR"" || goto MENU
:
kernel memdisk initrd=memtest.bin iso raw
...

See there are two lines that seem to miss some information. When I compare this to what I have on my test system it looks like the name of this item is somehow lost. Please check in your FOG web UI -> FOG Configuration -> iPXE Menu Item Settings: Do you have a section called fog.memtest there? I suppose you haven’t. Not exactly sure where it went but I can imagine this being part of your problem.

Try to manually re-create that menu item (FOG Configuration -> iPXE New Menu Entry).

Menu Item fog.memtest
Description Run Memtest86+
Parameters <LEAVE THAT EMPTY!>
Boot Options <LEAVE THAT EMPTY!>
Menu Show with All Hosts

Richardpofnz

@Sebastian-Roth said in Deployment and Quick Delete Host not working from iPXE menu:

When I compare this to what I have on my test system it looks like the name of this item is somehow lost. Please

Hey Sebastian,

I mentioned earlier but I removed the memtest menu item as it wasn’t required. Also to note, I was encountering this same issue before I removed it as well. I’ve added it once again as you’ve instructed and it is still behaving the same way.

The other thing I changed was the boot exit type for BIOS from SANBOOT to GRUB_FIRST_FOUND_WINDOWS as some other HP Servers were giving me an error with SANBOOT. The workstations experiencing this issue are EFI and still using the default REFIND_EFI exit type.

Updated output:

#!ipxe
set fog-ip 192.168.25.16
set fog-webroot fog
set boot-url http://${fog-ip}/${fog-webroot}
cpuid --ext 29 && set arch x86_64 || set arch i386
goto get_console
:console_set
colour --rgb 0x00567a 1 ||
colour --rgb 0x00567a 2 ||
colour --rgb 0x00567a 4 ||
cpair --foreground 7 --background 2 2 ||
goto MENU
:alt_console
cpair --background 0 1 ||
cpair --background 1 2 ||
goto MENU
:get_console
console --picture http://192.168.25.16/fog/service/ipxe/bg.png --left 100 --right 80 && goto console_set || goto alt_console
:MENU
menu
colour --rgb 0x00567a 0 ||
cpair --foreground 1 1 ||
cpair --foreground 0 3 ||
cpair --foreground 4 4 ||
item --gap Host is registered as MWAT-001!
item --gap -- -------------------------------------
item fog.local Boot from hard disk
item
item fog.keyreg Update Product Key
item fog.deployimage Deploy Image
item fog.multijoin Join Multicast Session
item fog.quickdel Quick Host Deletion
item fog.sysinfo Client System Information (Compatibility)
item fog.memtest Run Memtest86+
choose --default fog.local --timeout 30000 target && goto ${target}
:fog.local
chain -ar ${boot-url}/service/ipxe/grub.exe --config-file="find --set-root /BOOTMGR;chainloader /BOOTMGR"" || goto MENU
:
kernel memdisk initrd=memtest.bin iso raw
initrd memtest.bin
boot || goto MENU
:fog.keyreg
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param keyreg 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.deployimage
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param qihost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.multijoin
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param sessionJoin 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.quickdel
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param delhost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.sysinfo
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=127000 web=http://192.168.25.16/fog/ consoleblank=0 rootfstype=ext4 storage=192.168.25.16:/images/ storageip=192.168.25.16 loglevel=4 mode=sysinfo
imgfetch init_32.xz
boot || goto MENU
:fog.memtest
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=127000 web=http://192.168.25.16/fog/ consoleblank=0 rootfstype=ext4 storage=192.168.25.16:/images/ storageip=192.168.25.16 loglevel=4
imgfetch init_32.xz
boot || goto MENU
:bootme
chain -ar http://192.168.25.16/fog/service/ipxe/boot.php##params ||
goto MENU
autoboot

Sebastian Roth

@Richardpofnz said:

I’ve added it once again as you’ve instructed and it is still behaving the same way.

From the new output you posted it doesn’t look much better. There is a new menu item fog.memtest at the bottom but it’s not what I’d like it to look like. It’s been a fair while since I last dug through that boot menu generation code and I don’t have all the details in my head. Looking at the code again I figured that the memdisk menu entry is hardcoded to DB id 2 - not very nice but I am sure this was done for a reason some years ago. So I was wrong when I suggested to just re-add it manually. You’d probably need to recreate that table of the database to get it back.

I understand that you intentionally got rid of it and that you say the issue has been there before deleting it. But I am still wondering if this is playing a role in this case.

The workstations experiencing this issue are EFI and still using the default REFIND_EFI exit type.

Just saw this… Do I get this right?!? Are you saying that this problem with “Deploy Image” and “Quick Host Deletion” only happen on EFI machines?!? Most probably I just got wrong what you said here.

Richardpofnz

@Sebastian-Roth

Yes this is correct.

The issue only occurs on EFI machines but works as expected on BIOS.

Here is an example of it working on a BIOS machine:

Sebastian Roth

@Richardpofnz said in Deployment and Quick Delete Host not working from iPXE menu:

The issue only occurs on EFI machines but works as expected on BIOS.

Well that’s the most interesting and important information on this topic!! @george1421 have you seen UEFI clients behaving differently on the iPXE menu/login/params stuff yet?

@Richardpofnz Do you have diffeent UEFI model machines or is it all the same ones?

george1421

@Sebastian-Roth said in Deployment and Quick Delete Host not working from iPXE menu:

ave you seen UEFI clients behaving differently on the iPXE menu/login/params stuff yet

No I haven’t. This is a unique case. So both Deploy Image (quick image) and Quick Delete function by sending a single parameter (delhost or qihost) to boot.php. Its the same for the registration commands too. But registration works. I haven’t looked at the boot.php flow but what might make FOS fall through both qihost and delhost to end up executing the compat test code.

Possibly we could set a global kernel parameter of isdebug=yes then get this menu to appear by picking delhost or qihost. At them menu break out and confirm that the kernel parameters sent from FOG are correct. If they are correct then we could focus on the FOS code (/bin/fog) to why it might fail through to compat test mode. When testing is done then isdebug=yes should be removed so FOG runs correctly.

Richardpofnz

@Sebastian-Roth

Hi Sebastian,

Both test workstations I’ve been using have the same hardware. Motherboard model - ASRock Fatal1ty X399.

I’ve just tried replicating on a UEFI Hyper-V VM booting to ipxe.efi and can confirm this works as expected! Looks like the problem is hardware specific.

Definitely not a deal breaker as a manual host registration works and the capture/deploy command from web interface is working perfectly.

Thanks
Richard

Sebastian Roth

@Richardpofnz Sorry for the delay. Had to sort many other things.

Let’s try to figure out which part of iPXE exactly is causing this. For that I suggest you add a new iPXE menu item: web UI -> FOG Configuration -> iPXE New Menu Entry:

Menu Item - fog.ipxeshell
Description - iPXE shell
Parameters - shell
Menu Show with - All Hosts

Now boot up one of those hosts that has the issue and select the menu item “iPXE shell” and try this:

iPXE> echo ${username}

iPXE> echo ${password}

iPXE> login

# type in username and password

iPXE> echo ${username}
fog
iPXE> echo ${password}
password

As you see in my example the variables are empty at first but after typing those into the login dialog the variables should be set accordingly. Does this work on your ASRock Fatal1ty X399 UEFI machines?

Richardpofnz

@Sebastian-Roth

Hi Sebastian,

No problems on the delay.

Yes this appears to be working as expected:

Sebastian Roth

@Richardpofnz Ok let’s see what the server side does. On the FOG server run tail -f /var/log/apache2/access.log (or tail -f /var/log/httpd/access_log if you are on CentOS/RHEL/fedora) while you do the menu stuff on the client.

On my test system it looks like this when I try “Deploy Image” from the menu:

192.168.2.10 - - [08/Jul/2019:14:11:10 +0200] "POST /fog/service/ipxe/boot.php HTTP/1.1" 200 2768 "-" "iPXE/1.0.0+ (9907f)"
192.168.2.10 - - [08/Jul/2019:14:11:10 +0200] "GET /fog/service/ipxe/bg.png HTTP/1.1" 200 21280 "-" "iPXE/1.0.0+ (9907f)"
192.168.2.10 - - [08/Jul/2019:14:11:31 +0200] "POST /fog/service/ipxe/boot.php HTTP/1.1" 200 813 "-" "iPXE/1.0.0+ (9907f)"
192.168.2.10 - - [08/Jul/2019:14:11:34 +0200] "POST /fog/service/ipxe/boot.php HTTP/1.1" 200 464 "-" "iPXE/1.0.0+ (9907f)"
192.168.2.10 - - [08/Jul/2019:14:11:35 +0200] "POST /fog/service/ipxe/boot.php HTTP/1.1" 200 481 "-" "iPXE/1.0.0+ (9907f)"
192.168.2.10 - - [08/Jul/2019:14:11:35 +0200] "GET /fog/service/ipxe/bzImage HTTP/1.1" 200 8389280 "-" "iPXE/1.0.0+ (9907f)"
...

Deployment and Quick Delete Host not working from iPXE menu

128

12.1k

17.3k

155.3k