PXE Boot not working properly from Storage Node

george1421 · Feb 19, 2020, 1:45 PM

@Silv4n Will you collect some information for me?

At the main site dhcp, what are the settings for options 66 and 67
At the remote site dhcp what are the settings for the options 66 and 67.

At the remote site for the FOG storage node. Post the content of the /tftpboot/default.ipxe file.

Key in the following URL once from the main fog server and once from the remote storage node.

http://<ip_address>/fog/service/ipxe/boot.php

This will give you a screen full of text. The text represents the fog boot menu. There is something going on that we don’t understand at the moment.

Silv4n · Feb 19, 2020, 1:52 PM

@george1421
Main site: It’s currently the DMZ, so we don’t actually use PXE Boot there
Remote site: 10.144.1.22, undionly.kpxe

10.144.1.22 remote site /tftpboot/default.ipxe

#!ipxe
cpuid --ext 29 && set arch x86_64 || set arch ${buildarch}
params
param mac0 ${net0/mac}
param arch ${arch}
param platform ${platform}
param product ${product}
param manufacturer ${product}
param ipxever ${version}
param filename ${filename}
param sysuuid ${uuid}
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
:bootme
chain http://10.144.1.22/fog/service/ipxe/boot.php##params

10.144.1.22 boot.php (remote)

#!ipxe
set fog-ip 10.51.1.104
set fog-webroot fog
set boot-url http://${fog-ip}/${fog-webroot}
cpuid --ext 29 && set arch x86_64 || set arch i386
goto get_console
:console_set
colour --rgb 0x00567a 1 ||
colour --rgb 0x00567a 2 ||
colour --rgb 0x00567a 4 ||
cpair --foreground 7 --background 2 2 ||
goto MENU
:alt_console
cpair --background 0 1 ||
cpair --background 1 2 ||
goto MENU
:get_console
console --picture http://10.51.1.104/fog/service/ipxe/bg.png --left 100 --right 80 && goto console_set || goto alt_console
:MENU
menu
colour --rgb 0xff0000 0 ||
cpair --foreground 1 1 ||
cpair --foreground 0 3 ||
cpair --foreground 4 4 ||
item --gap Host is NOT registered!
item --gap -- -------------------------------------
item fog.local Boot from hard disk
item fog.memtest Run Memtest86+
item fog.reginput Perform Full Host Registration and Inventory
item fog.reg Quick Registration and Inventory
item fog.deployimage Deploy Image
item fog.multijoin Join Multicast Session
item fog.sysinfo Client System Information (Compatibility)
choose --default fog.local --timeout 3000 target && goto ${target}
:fog.local
sanboot --no-describe --drive 0x80 || goto MENU
:fog.memtest
kernel memdisk initrd=memtest.bin iso raw
initrd memtest.bin
boot || goto MENU
:fog.reginput
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 web=http://10.51.1.104/fog/ consoleblank=0 rootfstype=ext4 storage=10.51.1.104:/images/ storageip=10.51.1.104 loglevel=4 mode=manreg
imgfetch init_32.xz
boot || goto MENU
:fog.reg
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 web=http://10.51.1.104/fog/ consoleblank=0 rootfstype=ext4 storage=10.51.1.104:/images/ storageip=10.51.1.104 loglevel=4 mode=autoreg
imgfetch init_32.xz
boot || goto MENU
:fog.deployimage
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param qihost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.multijoin
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param sessionJoin 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.sysinfo
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 web=http://10.51.1.104/fog/ consoleblank=0 rootfstype=ext4 storage=10.51.1.104:/images/ storageip=10.51.1.104 loglevel=4 mode=sysinfo
imgfetch init_32.xz
boot || goto MENU
:bootme
chain -ar http://10.51.1.104/fog/service/ipxe/boot.php##params ||
goto MENU
autoboot

main site 10.51.1.104 boot.php (It’s HTTPS, I’m not sure if that makes an difference)

#!ipxe
set fog-ip 10.51.1.104
set fog-webroot fog
set boot-url https://${fog-ip}/${fog-webroot}
cpuid --ext 29 && set arch x86_64 || set arch i386
goto get_console
:console_set
colour --rgb 0x00567a 1 ||
colour --rgb 0x00567a 2 ||
colour --rgb 0x00567a 4 ||
cpair --foreground 7 --background 2 2 ||
goto MENU
:alt_console
cpair --background 0 1 ||
cpair --background 1 2 ||
goto MENU
:get_console
console --picture https://10.51.1.104/fog/service/ipxe/bg.png --left 100 --right 80 && goto console_set || goto alt_console
:MENU
menu
colour --rgb 0xff0000 0 ||
cpair --foreground 1 1 ||
cpair --foreground 0 3 ||
cpair --foreground 4 4 ||
item --gap Host is NOT registered!
item --gap -- -------------------------------------
item fog.local Boot from hard disk
item fog.memtest Run Memtest86+
item fog.reginput Perform Full Host Registration and Inventory
item fog.reg Quick Registration and Inventory
item fog.deployimage Deploy Image
item fog.multijoin Join Multicast Session
item fog.sysinfo Client System Information (Compatibility)
choose --default fog.local --timeout 3000 target && goto ${target}
:fog.local
sanboot --no-describe --drive 0x80 || goto MENU
:fog.memtest
kernel memdisk initrd=memtest.bin iso raw
initrd memtest.bin
boot || goto MENU
:fog.reginput
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 web=https://10.51.1.104/fog/ consoleblank=0 rootfstype=ext4 storage=10.51.1.104:/images/ storageip=10.51.1.104 loglevel=4 mode=manreg
imgfetch init_32.xz
boot || goto MENU
:fog.reg
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 web=https://10.51.1.104/fog/ consoleblank=0 rootfstype=ext4 storage=10.51.1.104:/images/ storageip=10.51.1.104 loglevel=4 mode=autoreg
imgfetch init_32.xz
boot || goto MENU
:fog.deployimage
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param qihost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.multijoin
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param sessionJoin 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.sysinfo
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=275000 web=https://10.51.1.104/fog/ consoleblank=0 rootfstype=ext4 storage=10.51.1.104:/images/ storageip=10.51.1.104 loglevel=4 mode=sysinfo
imgfetch init_32.xz
boot || goto MENU
:bootme
chain -ar https://10.51.1.104/fog/service/ipxe/boot.php##params ||
goto MENU
autoboot

george1421 · Feb 19, 2020, 2:01 PM

@Silv4n The root of the issue is the main site uses https and the remote sites are using http.

Silv4n · Feb 19, 2020, 2:05 PM

@george1421 Ok, I’ve actually even tried to reinstall the storage node with https, but it gave me an error. Can I use a master node instead, which still uses the db etc. on the main server or should the storage work with https?

george1421 · Feb 19, 2020, 2:11 PM

@Silv4n This is an interesting problem. I think we will need to ping the @developers on this one. I’m pretty sure we can just copy over the apache stuff from the master node to the storage node to enable https, on the storage node. The issue is telling the storage node to use https in its scripting vs http.

Silv4n · Feb 19, 2020, 2:15 PM

@george1421 I’m gonna assume, that I can’t just change the boot.php file from the remote site to use https, right?

george1421 · Feb 19, 2020, 2:18 PM

@Silv4n There is a setting some place where it defines which protocol to use. There is a text based config file in the fog www directory.

george1421 · Feb 19, 2020, 8:25 AM

@george1421 It would be interesting if you copied over the apache certificates from the master node to the storage node as well as the apache configuration required to make the storage node https compatible. THEN in the storage node configuration there is a web url setting where we need to add in https.

This is just me thinking without any input from the developers.

Silv4n · Feb 19, 2020, 2:27 PM

@george1421 Hmm, okay thanks for your input, I guess I’ll wait till the devs can confirm this, before breaking the storage node or something, because the sync for example works without any issues.

Silv4n · Feb 19, 2020, 2:30 PM

@george1421 Nvm, I’m gonna try

george1421 · Feb 19, 2020, 2:31 PM

@Silv4n said in PXE Boot not working properly from Storage Node:

Nvm, I’m gonna try

That was my thought, it already doesn’t work. How bad could you make it?

Silv4n · Feb 19, 2020, 2:49 PM

@george1421 I haven’t copied the certs over yet, because there were already certs on the server, but now this error appears in the apache log, I’m not sure if copying over solves that:
http://prntscr.com/r4nmiu

george1421 · Feb 19, 2020, 8:53 AM

@Silv4n Your link only contains an ugly picture of the US president in the ad. Please do scare me like that this early in the morning. I’m a USA citizen I see enough of that here…

Silv4n · Feb 19, 2020, 2:55 PM

@george1421 Based on cookies i guess ;), for me it shows the screenshot, but im gonna copy it here instead:

[Wed Feb 19 15:43:44.859522 2020] [ssl:emerg] [pid 16345] AH02565: Certificate and private key 10.144.1.22:443:0 from /var/www/fog/management/other/ssl/srvpublic.crt and /opt/fog/snapins/ssl/.srvprivate.key do not match

george1421 · Feb 19, 2020, 8:57 AM

@Silv4n I would save those existing keys off to the side and then copy the keys over from the main fog server. This is not ‘technically’ the right way, but we just need to see it work right now.

The right way would be to build new keys for the storage node using the root CA created on the master FOG node.

Silv4n · Feb 19, 2020, 3:00 PM

@george1421 Alright, I’m gonna test that tomorrow morning or tonight and than I’ll leave feedback here.

Silv4n · Feb 20, 2020, 7:50 AM

@george1421 Alright, I’ve copied over the

/opt/fog/snapins/ssl/.srvprivate.key

And the apache service started now and after a reinstall with HTTPS (which worked now without an issues) I can access the Web GUI of the Storage Node via HTTPS. It also generated a new boot file etc. However, the PXE Boot still throws the chainloading error, when trying to actually boot in to something. It also tries to connect there still with HTTP.

Sebastian Roth · Feb 20, 2020, 3:47 AM

@Silv4n said in PXE Boot not working properly from Storage Node:

It also tries to connect there still with HTTP

Edit /tftpboot/default.ipxe on the storage node and adjust the URL in the last line.

Though there is another thing that you need to fix I’d guess. I haven’t done a HTTPS enabled storage node setup in a while. But I’d think your iPXE binaries on the storage node do not include the correct cert yet.

As there was an issue in the build script of FOG 1.5.7 I’d suggest you do the following on your storage node:

Make sure you have the whole CA copied form your master node to your storage node. It’s in /opt/fog/snapins/ssl/CA/ and includes hidden files, so make sure you grab all of it. Put that in the same location on the storage node and make sure ownership and rights are set exactly as they were before (compare ls -al output).
Grab the iPXE build script from the latest FOG project development code branch and rebuild the iPXE binaries to include your CA cert:

sudo su -
cd /path/to/your/fogproject-source-dir/
cd utils/FOGiPXE
wget -O buildipxe.sh https://raw.githubusercontent.com/FOGProject/fogproject/dev-branch/utils/FOGiPXE/buildipxe.sh
chmod +x buildipxe.sh
./buildipxe.sh

Keep an eye on this to be sure it doesn’t end with an error. Then copy the new binaries over to the destination:

cd ../../packages/tftp/
mkdir /tftpboot/arm64-efi
mkdir /tftpboot/10secdelay/arm64-efi
mkdir /tftpboot/10secdelay/i386-efi
find -type f -exec cp -Rfv {} /tftpboot/{} \;

Make sure you edit /opt/fog/.fogproject and set httpproto='https' for when you re-run the FOG installer in the future.

I know this might seem overly complicated but from my point of view those steps are best suited in your current situation of half HTTP/HTTPS.

Silv4n · Feb 20, 2020, 10:11 AM

@Sebastian-Roth Ok, in /tftpboot/default.ipxe there currently is chain https://10.144.1.22/fog/service/ipxe/boot.php##params, so there is already https. Im’m gonna try now the cert gen etc.

Silv4n · Feb 20, 2020, 10:49 AM

@Sebastian-Roth So I’ve generated the new certs etc. and now I have a new error message (Permission denied):
https://imgur.com/a/yK4TwNZ

PXE Boot not working properly from Storage Node

180

12.0k

17.3k

155.2k