Not able to TFTP boot. Invalid Argument Error

[hancocza]

@george1421 Correct. In the past, since I started using HTTPS, I’ve had to change the default.ipxe file to point to http vs https. https never worked for me (I’m guessing because i never compiled the binaries after updating). So I’ve always changed the chain address to be http instead of https. Now that doesn’t work either.

[george1421]

@hancocza Ok just to be clear you haven’t changed anything in FOG’s default.ipxe other than http and https?

What do you get if you key the following into a browser?

http://<fog_server_ip>/fog/service/ipxe/boot.php?mac=00:00:00:00:00:01

That should give you a screen of text, which is the FOG ipxe menu.

[hancocza]

@george1421 said in Not able to TFTP boot. Invalid Argument Error:

http://<fog_server_ip>/fog/service/ipxe/boot.php?mac=00:00:00:00:00:01

Here is what I get. I do change the IP section from a FQDN to the IP. When installing I put the FQDN as the IP address so that I don’t have to change too much.

#!ipxe
set fog-ip xxx.xxx.xxx.xxx
set fog-webroot fog
set boot-url https://${fog-ip}/${fog-webroot}
cpuid --ext 29 && set arch x86_64 || set arch i386
goto get_console
:console_set
colour --rgb 0x00567a 1 ||
colour --rgb 0x00567a 2 ||
colour --rgb 0x00567a 4 ||
cpair --foreground 7 --background 2 2 ||
goto MENU
:alt_console
cpair --background 0 1 ||
cpair --background 1 2 ||
goto MENU
:get_console
console --picture https://xxx.xxx.xxx.xxx/fog/service/ipxe/bg.png --left 100 --right 80 && goto console_set || goto alt_console
:MENU
menu
colour --rgb 0xff0000 0 ||
cpair --foreground 1 1 ||
cpair --foreground 0 3 ||
cpair --foreground 4 4 ||
item --gap Host is NOT registered!
item --gap -- -------------------------------------
item fog.local Boot from hard disk
item fog.memtest Run Memtest86+
item fog.reginput Perform Full Host Registration and Inventory
item fog.reg Quick Registration and Inventory
item fog.deployimage Deploy Image
item fog.multijoin Join Multicast Session
item fog.sysinfo Client System Information (Compatibility)
choose --default fog.local --timeout 3000 target && goto ${target}
:fog.local
sanboot --no-describe --drive 0x80 || goto MENU
:fog.memtest
kernel memdisk initrd=memtest.bin iso raw
initrd memtest.bin
boot || goto MENU
:fog.reginput
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=127000 web=https://xxx.xxx.xxx.xxx/fog/ consoleblank=0 rootfstype=ext4 storage=xxx.xxx.xxx.xxx:/images/ storageip=xxx.xxx.xxx.xxx loglevel=4 mode=manreg
imgfetch init_32.xz
boot || goto MENU
:fog.reg
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=127000 web=https://xxx.xxx.xxx.xxx/fog/ consoleblank=0 rootfstype=ext4 storage=xxx.xxx.xxx.xxx:/images/ storageip=xxx.xxx.xxx.xxx loglevel=4 mode=autoreg
imgfetch init_32.xz
boot || goto MENU
:fog.deployimage
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param qihost 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.multijoin
login
params
param mac0 ${net0/mac}
param arch ${arch}
param username ${username}
param password ${password}
param sessionJoin 1
isset ${net1/mac} && param mac1 ${net1/mac} || goto bootme
isset ${net2/mac} && param mac2 ${net2/mac} || goto bootme
param sysuuid ${uuid}
:fog.sysinfo
kernel bzImage32 loglevel=4 initrd=init_32.xz root=/dev/ram0 rw ramdisk_size=127000 web=https://xxx.xxx.xxx.xxx/fog/ consoleblank=0 rootfstype=ext4 storage=xxx.xxx.xxx.xxx:/images/ storageip=xxx.xxx.xxx.xxx loglevel=4 mode=sysinfo
imgfetch init_32.xz
boot || goto MENU
:bootme
chain -ar https://xxx.xxx.xxx.xxx/fog/service/ipxe/boot.php##params ||
goto MENU
autoboot

[george1421]

@hancocza OK that tells us that boot.php code is working on the fog server. That is the complete iPXE boot menu. So we know its not the fog server causing you pain.

So you haven’t changed the fog default.ipxe, boot.php is working because the menu is created when you call it from a browser.

so the question is what is being passed that is causing boot.php to return an invalid ipxe menu.

The next step is to usb boot again and get the error. Then inspect the apache access log, error log, and php-fpm error log files. These are typically in /var/log directory.

[george1421]

@george1421 Just wondering is this fog client computer you are trying to USB boot registered in FOG? If so delete the registration and see if it throws the same error.

[Sebastian Roth]

@hancocza Your picture shows the iPXE error http://ipxe.org/err/1c0de8 - see this is definitely talking about TLS being the issue. Now having a closer look at the iPXE menu output you posted from “boot.php” I see that i does a chain https://... right at the end as well as console --picture https://... and others which all will likely fail if you don’t have iPXE binaries compiled with your certificate in it.

Those URLs are generated and the http(s) part is derived from the client request itself. So if a client asks for http:// all the URLs in the bootmenu will also be http:// - but we generate an apache config to redirect HTTP to HTTPS and therefore you end up with https:// URLs in the iPXE boot menu. Either you disable the forced HTTPS redirect or you look into compiling correct iPXE binaries. The later should be real easy using the script as I suggested.

[hancocza]

@Sebastian-Roth I ran the recompiling script last friday. At the end it says it needs to go through the FOG installation again, so i did. After that, it still gave the same error. Should I be allowing it to go through the FOG installation again or does that basically reset the iPXE binaries again?

[hancocza]

@george1421 said in Not able to TFTP boot. Invalid Argument Error:

@george1421 Just wondering is this fog client computer you are trying to USB boot registered in FOG? If so delete the registration and see if it throws the same error.

This happened on both laptops that were registered as well as a few that were not.

I also checked all of the logs after attempting to boot, and it didn’t update any of those logs. I assume that means it’s in line with what Sebastian said about the TLS error.

[Sebastian Roth]

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

Should I be allowing it to go through the FOG installation again or does that basically reset the iPXE binaries again?

When you run the buidlipxe.sh Script it compiles new binaries including the correct SSL cert and puts them in the “installer directory”. From now on you can run the FOG installer as often as you want, it should always install those HTTPS-enabled iPXE binaries!

[hancocza]

@Sebastian-Roth Hmm… I did that and still get the same error after. Same error code as well, pointing at the TLS handshake.

[Sebastian Roth]

@hancocza Sure you don’t get any error from the iPXE build script?

Do you use a custom SSL certificate for the FOG web server?

[hancocza]

@Sebastian-Roth I don’t think i get one from the script. How could i check?

I do use a custom SSL cert for the web server. In the apache2 site config, it points to the correct cert and key. That’s also why I get the password prompts during installation.

[Sebastian Roth]

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

I do use a custom SSL cert for the web server.

Ah sorry, I missed that. Then you need to adjust the utils/FOGiPXE/buildipxe.sh Script to point to those custom certs as well. Edit that file and put in the correct paths for CA cert and server cert:

#!/bin/bash
BUILDOPTS="TRUST=/var/www/fog/management/other/ca.cert.pem CERT=/var/www/fog/management/other/ca.cert.pem"
IPXEGIT="https://git.ipxe.org/ipxe.git"
...

This is the default. Change to where you have your custom cert files and compile again! Then rerun the installer as well.

[hancocza]

@Sebastian-Roth Tried that with our cert that we use. I replaced both the path for both TRUST= and CERT= with the certificate path, compiled and ran the installer. I tried it with the default.ipxe unchanged, as well as with the IP in place of the FQDN in the chain line.

My buildipxe.sh looks like:

#!/bin/bash
BUILDOPTS=“TRUST=/home/fogserver/Documents/fogcert.crt CERT=/home/fogserver/Documents/fogcert.crt”
IPXEGIT=“https://git.ipxe.org/ipxe.git”

The chain line of the default.ipxe looks like this:

chain https://xxx.xxx.xxx.xxx/fog/service/ipxe/boot.php##params

Am I messing up the placement of the certificate? Apache asks for three separate certificate components (certificate, key, chain), am I supposed to have two different certificates specified in the buildipxe.sh?
…

[Sebastian Roth]

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

am I supposed to have two different certificates specified in the buildipxe.sh?

Yes! The one going in TRUST is the CA cert or possibly what you use in Chain parameter in the apache config. If it is a real chain of two or more certs in the chain I am not sure if iPXE can handle it this way. Give it a try. The CERT is just that, the webserver certificate.

By the way, keeping the certs in the home directory might not be a wise idea. Someone comes along and cleans up the home dir one day and boom all is gone.

[hancocza]

@Sebastian-Roth I’m kind of at a loss on this. I have tried many different combinations of the certs that I have, have tried using the one fog uses, and have tried the ones provided by iPXE. They all continue to give me invalid argument error. I added the DEBUG=tls line to the buildipxe.sh file, and tried booting. It said my cert was added to the certstore, and then ran through what I’m guessing are handshakes. In the end I still get the invalid argument error.

Is there a way to use my ssl certs for just the web server? Then all of the fog functions would use the supplied fog certs?

[Sebastian Roth]

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

Is there a way to use my ssl certs for just the web server?

Sorry, but no. iPXE requests information through HTTP(S) and you can’t separate that from the web interface (certs). I suppose you could setup another webserver (like nginx) on another port and run the FOG web UI on that. But that’s not a great way - don’t go there!

I am fairly sure we can get this to work for you. Let’s start by talking about the certificate. Where did you get it from. What CN is it dedicated to and does it have any extensions enabled? Please run openssl x509 -in /path/to/custom/certificate.crt -text -noout and post output here. A certificate is the public part of a public/private key crypto system and hence there should be no issue posting this content here. If you’d prefer you could also just send me a private message with the information.

[hancocza]

@Sebastian-Roth
Darn.

The certificate was purchase through Go Daddy. The CN is associated with our main web server. I added the domain name of the fog server to the “other names” section in order to cover that as well. I don’t believe it has any extensions enabled.

Here is the output:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1807900440220026086 (0x1916f3df289644e6)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
        Validity
            Not Before: Mar 15 18:52:13 2018 GMT
            Not After : Jul 13 16:53:00 2019 GMT
        Subject: OU = Domain Control Validated, CN = csims.clas.gvsu.edu
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:e2:79:a6:3f:ac:83:5a:ec:97:ab:2c:74:95:a5:
                    2e:cc:30:41:f6:32:f0:4f:e5:2a:fb:c7:dd:7d:52:
                    9b:a0:8c:20:4e:1d:c2:7f:d5:99:ca:3b:b7:f5:ca:
                    05:dc:f6:85:a8:e5:99:03:95:77:6b:49:67:fe:b9:
                    cb:78:8c:da:9f:3b:89:db:46:7a:c7:e2:ed:22:04:
                    84:f5:61:2f:58:9d:0a:ee:66:9e:26:40:fe:54:8f:
                    a8:44:fd:75:16:dd:1a:24:d5:77:28:8f:f5:79:76:
                    ab:9f:92:f2:fe:e1:f5:1e:17:e6:7f:d3:b2:07:52:
                    8f:60:94:28:3a:48:e6:8a:3b:57:0c:6d:4d:30:d3:
                    23:de:76:07:3a:f3:bf:60:ef:26:47:c4:17:45:54:
                    71:d7:ce:c0:e8:ef:c3:f8:42:d5:3c:47:1b:5d:97:
                    96:a6:2a:3d:dd:ac:d7:4e:38:03:68:f4:29:eb:80:
                    fb:48:04:40:f6:f7:4d:19:34:a5:d8:6e:ec:5b:15:
                    e1:97:42:17:4c:bc:c2:55:cf:44:80:ca:0d:5f:20:
                    fb:98:c6:25:e3:12:19:a0:bb:b2:e8:b1:cf:fc:2e:
                    00:10:ab:e6:7b:da:85:01:6d:0b:d7:ed:53:c8:ae:
                    d4:18:e4:52:ab:86:aa:c1:14:e3:6c:47:fe:0b:a2:
                    9c:db
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.godaddy.com/gdig2s1-815.crl

            X509v3 Certificate Policies: 
                Policy: 2.16.840.1.114413.1.7.23.1
                  CPS: http://certificates.godaddy.com/repository/
                Policy: 2.23.140.1.2.1

            Authority Information Access: 
                OCSP - URI:http://ocsp.godaddy.com/
                CA Issuers - URI:http://certificates.godaddy.com/repository/gdig2.crt

            X509v3 Authority Key Identifier: 
                keyid:40:C2:BD:27:8E:CC:34:83:30:A2:33:D7:FB:6C:B3:F0:B4:2C:80:CE

            X509v3 Subject Alternative Name: 
                DNS:csims.clas.gvsu.edu, DNS:www.csims.clas.gvsu.edu, DNS:fogserver.gvsu.edu, DNS:csimsweb.clas.gvsu.edu, DNS:viewlinc.clas.gvsu.edu
            X509v3 Subject Key Identifier: 
                AB:E1:0F:46:89:C4:69:F7:D0:6E:C6:A1:40:E4:C5:70:7A:EA:C4:74
    Signature Algorithm: sha256WithRSAEncryption
         2a:1a:42:5a:21:ba:ac:81:cd:6e:7b:6f:73:55:92:5b:cc:d5:
         93:de:32:7c:b7:56:53:d5:8c:7c:4c:5d:4b:6e:cd:d9:2c:8e:
         a4:87:39:9d:85:05:3f:c8:12:fc:c0:d2:b3:c8:de:67:15:02:
         b9:22:5a:6d:f1:6a:f7:12:a0:28:b8:e6:69:c6:82:c5:61:ce:
         ff:cf:a5:9c:f3:6c:08:51:04:c8:4f:8a:28:08:be:a4:06:d7:
         54:26:91:9f:3b:76:7f:cb:7c:71:63:e3:54:f0:d4:8a:f9:a2:
         06:cb:11:dd:a4:4c:5d:c1:9a:5d:bb:96:6f:13:90:56:e4:e2:
         bd:11:b2:83:67:c2:9f:99:9b:60:10:40:c8:8b:56:5c:3d:95:
         2d:24:d8:7d:53:2d:2f:eb:fe:73:c4:54:ff:fc:73:12:51:b4:
         86:16:64:56:bf:4c:99:96:d3:2e:0e:d5:33:58:84:09:6b:ce:
         16:f1:b8:91:2f:cd:8b:35:52:e7:3d:d1:83:b8:5e:d9:9b:ce:
         8c:f8:0f:80:5c:23:60:5a:91:07:45:e2:fc:8d:0c:ee:c6:56:
         8c:76:c4:23:33:14:e6:80:56:33:d1:ef:30:b1:26:be:9f:34:
         ac:b9:74:ae:9c:89:d5:b7:76:f0:cb:88:bb:7c:41:fc:d5:70:
         16:98:0a:a7

[Sebastian Roth]

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

Subject: OU = Domain Control Validated, CN = csims.clas.gvsu.edu

So this is the main subject it was created for. All the other DNS names we find in the alternative subject extension further down. I am not exactly sure about this from the top of my head but possibly iPXE does not support using the alternative subjects from a cert. I will check this out and get back to you.

[Sebastian Roth]

@hancocza Oh man/woman, I have overlooked the most obvious problem with this. iPXE is connecting to the IP address instead of hostname and therefore cannot match the names in the certificate. I just had a quick look at the manual pages and seems like iPXE can handle alternative subjects, so I was wrong in my last post.

Edit /tftpboot/default.ipxe on your FOG server and make it use the DNS name instead of the IP.