Not able to TFTP boot. Invalid Argument Error

hancocza

The certificate was purchase through Go Daddy. The CN is associated with our main web server. I added the domain name of the fog server to the “other names” section in order to cover that as well. I don’t believe it has any extensions enabled.

Here is the output:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1807900440220026086 (0x1916f3df289644e6)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
        Validity
            Not Before: Mar 15 18:52:13 2018 GMT
            Not After : Jul 13 16:53:00 2019 GMT
        Subject: OU = Domain Control Validated, CN = csims.clas.gvsu.edu
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:e2:79:a6:3f:ac:83:5a:ec:97:ab:2c:74:95:a5:
                    2e:cc:30:41:f6:32:f0:4f:e5:2a:fb:c7:dd:7d:52:
                    9b:a0:8c:20:4e:1d:c2:7f:d5:99:ca:3b:b7:f5:ca:
                    05:dc:f6:85:a8:e5:99:03:95:77:6b:49:67:fe:b9:
                    cb:78:8c:da:9f:3b:89:db:46:7a:c7:e2:ed:22:04:
                    84:f5:61:2f:58:9d:0a:ee:66:9e:26:40:fe:54:8f:
                    a8:44:fd:75:16:dd:1a:24:d5:77:28:8f:f5:79:76:
                    ab:9f:92:f2:fe:e1:f5:1e:17:e6:7f:d3:b2:07:52:
                    8f:60:94:28:3a:48:e6:8a:3b:57:0c:6d:4d:30:d3:
                    23:de:76:07:3a:f3:bf:60:ef:26:47:c4:17:45:54:
                    71:d7:ce:c0:e8:ef:c3:f8:42:d5:3c:47:1b:5d:97:
                    96:a6:2a:3d:dd:ac:d7:4e:38:03:68:f4:29:eb:80:
                    fb:48:04:40:f6:f7:4d:19:34:a5:d8:6e:ec:5b:15:
                    e1:97:42:17:4c:bc:c2:55:cf:44:80:ca:0d:5f:20:
                    fb:98:c6:25:e3:12:19:a0:bb:b2:e8:b1:cf:fc:2e:
                    00:10:ab:e6:7b:da:85:01:6d:0b:d7:ed:53:c8:ae:
                    d4:18:e4:52:ab:86:aa:c1:14:e3:6c:47:fe:0b:a2:
                    9c:db
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.godaddy.com/gdig2s1-815.crl

            X509v3 Certificate Policies: 
                Policy: 2.16.840.1.114413.1.7.23.1
                  CPS: http://certificates.godaddy.com/repository/
                Policy: 2.23.140.1.2.1

            Authority Information Access: 
                OCSP - URI:http://ocsp.godaddy.com/
                CA Issuers - URI:http://certificates.godaddy.com/repository/gdig2.crt

            X509v3 Authority Key Identifier: 
                keyid:40:C2:BD:27:8E:CC:34:83:30:A2:33:D7:FB:6C:B3:F0:B4:2C:80:CE

            X509v3 Subject Alternative Name: 
                DNS:csims.clas.gvsu.edu, DNS:www.csims.clas.gvsu.edu, DNS:fogserver.gvsu.edu, DNS:csimsweb.clas.gvsu.edu, DNS:viewlinc.clas.gvsu.edu
            X509v3 Subject Key Identifier: 
                AB:E1:0F:46:89:C4:69:F7:D0:6E:C6:A1:40:E4:C5:70:7A:EA:C4:74
    Signature Algorithm: sha256WithRSAEncryption
         2a:1a:42:5a:21:ba:ac:81:cd:6e:7b:6f:73:55:92:5b:cc:d5:
         93:de:32:7c:b7:56:53:d5:8c:7c:4c:5d:4b:6e:cd:d9:2c:8e:
         a4:87:39:9d:85:05:3f:c8:12:fc:c0:d2:b3:c8:de:67:15:02:
         b9:22:5a:6d:f1:6a:f7:12:a0:28:b8:e6:69:c6:82:c5:61:ce:
         ff:cf:a5:9c:f3:6c:08:51:04:c8:4f:8a:28:08:be:a4:06:d7:
         54:26:91:9f:3b:76:7f:cb:7c:71:63:e3:54:f0:d4:8a:f9:a2:
         06:cb:11:dd:a4:4c:5d:c1:9a:5d:bb:96:6f:13:90:56:e4:e2:
         bd:11:b2:83:67:c2:9f:99:9b:60:10:40:c8:8b:56:5c:3d:95:
         2d:24:d8:7d:53:2d:2f:eb:fe:73:c4:54:ff:fc:73:12:51:b4:
         86:16:64:56:bf:4c:99:96:d3:2e:0e:d5:33:58:84:09:6b:ce:
         16:f1:b8:91:2f:cd:8b:35:52:e7:3d:d1:83:b8:5e:d9:9b:ce:
         8c:f8:0f:80:5c:23:60:5a:91:07:45:e2:fc:8d:0c:ee:c6:56:
         8c:76:c4:23:33:14:e6:80:56:33:d1:ef:30:b1:26:be:9f:34:
         ac:b9:74:ae:9c:89:d5:b7:76:f0:cb:88:bb:7c:41:fc:d5:70:
         16:98:0a:a7

Sebastian Roth

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

Subject: OU = Domain Control Validated, CN = csims.clas.gvsu.edu

So this is the main subject it was created for. All the other DNS names we find in the alternative subject extension further down. I am not exactly sure about this from the top of my head but possibly iPXE does not support using the alternative subjects from a cert. I will check this out and get back to you.

Sebastian Roth

@hancocza Oh man/woman, I have overlooked the most obvious problem with this. iPXE is connecting to the IP address instead of hostname and therefore cannot match the names in the certificate. I just had a quick look at the manual pages and seems like iPXE can handle alternative subjects, so I was wrong in my last post.

Edit /tftpboot/default.ipxe on your FOG server and make it use the DNS name instead of the IP.

hancocza

@Sebastian-Roth the default.ipxe file already specifies the fqdn of fogserver.gvsu.edu. on 1.5.0, I would just change it to the IP and change https to http and not recompile, and it all worked fine. Since we’ve been doing this, I’ve left it as https://fogserver.gvsu.edu/…

If it could still be an issue of the two separate certs (ca and server) in the buildipxe.sh, I’m not sure what to use as the ca one. Go Daddy uses an intermediate cert instead of a CA, but that was the chain one that we weren’t sure if it’d work.

Sebastian Roth

@hancocza Please check to see if you have also set the FQDN in FOG configuration -> FOG settings -> Web Server -> WEB HOST! Writing my last messages in the train where I don’t have good access to the code I forgot about this.

From the information you posted I would expect you’d need to use the GoDaddy Secure Server Certificate (Intermediate Certificate) - G2 as TRUST. This (or more precisely its private key) should be exactly the one that was used to sign your certificate.

hancocza

@Sebastian-Roth said in Not able to TFTP boot. Invalid Argument Error:

@hancocza Please check to see if you have also set the FQDN in FOG configuration -> FOG settings -> Web Server -> WEB HOST! Writing my last messages in the train where I don’t have good access to the code I forgot about this.

From the information you posted I would expect you’d need to use the GoDaddy Secure Server Certificate (Intermediate Certificate) - G2 as TRUST. This (or more precisely its private key) should be exactly the one that was used to sign your certificate.

I had the IP set in the WEB HOST field. I switched that now. Also updated the TRUST parameter in buildipxe.sh to have that cert. I’ll test tomorrow and see what happens!

hancocza

@Sebastian-Roth Just tested, still a no go. Trust line looks like this with the intermediate cert:

BUILDOPTS=“CERT=/home/fogserver/Documents/csims.clas.gvsu.edu/fogcert.pem TRUST=/home/fogserver/Documents/csims.clas.gvsu.edu/GoDaddyCA.pem”

Sebastian Roth

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

GoDaddyCA.pem

Sure this is the same one that I pointed out? Please run the following commands to check:

cd /tmp/
wget https://certs.godaddy.com/repository/gdig2.crt.pem
sha256sum gdig2.crt.pem /home/fogserver/Documents/csims.clas.gvsu.edu/GoDaddyCA.pem
...

See if you get the same hash for both files.

You might need to use different ones for iPXE versus the ones used for Apache. Apache needs the full trust chain I reckon. You have some major root CA certificates installed on your system already and point Apache to some chain certificates (possibly two intermediate ones) so the chain is fully trusted. For iPXE I guess (sorry, don’t have enough time to test or look it up right now) you only need to trust the one single certificate/key that was used to issue your web server certificate and I think that is https://certs.godaddy.com/repository/gdig2.crt.pem in your case.

hancocza

@Sebastian-Roth It wasn’t the one you mentioned. I downloaded the correct one and put it in place, still got an invalid argument error. Do i need to append my certificate to the trust as well, so that it looks like:

TRUST=/home/fogserver/Desktop/gdig2.crt.pem,/home/fogserver/Documents/csims.clas.gvsu.edu/fogcert.crt

I was looking at this website when I came across that: https://ipxe.org/crypto

Sebastian Roth

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

Do i need to append my certificate to the trust as well

Don’t think you have to. But reading more about how (open)ssl certificate verification works I figured that I was wrong with one of my assumptions. The intermediate certificate is not enough as it is not selfiestick signed (only root certs are). So you’d need to specify intermediate and root cert as TRUST.

Sebastian Roth

@hancocza Try using TRUST=gdroot-g2.crt.pem,gdig2.crt.pem

https://certs.godaddy.com/repository/gdroot-g2.crt
https://certs.godaddy.com/repository/gdig2.crt.pem

I think you need to convert the gdroot-g2.crt to PEM format. I read that iPXE can only handle PEM but not DER cert format. Convert using openssl x509 -inform DER -outform PEM -text -in gdroot-g2.crt -out gdroot-g2.crt.pem

You might want to check the whole chain using openssl as well: openssl verify -CAfile gdroot-g2.crt.pem -untrusted gdig2.crt.pem fogcert.pem

all untested…

EDIT: Posted on the iPXE developers mailinglist: http://lists.ipxe.org/pipermail/ipxe-devel/2018-December/006395.html

hancocza

@Sebastian-Roth Looks like i didn’t need to convert the root certificate, it was already in pem format. I ran the openssl verify command, and it returned an ‘OK’. I’ll insert that into the trust and recompile. I wont be able to test until tomorrow morning, so I’ll update you then. Thanks for all the help!

hancocza

@Sebastian-Roth Just tested, still the same. I ran the certstat command in the ipxe shell and it only listed the fogcert.crt certificate. Should it also have the other ones listed?

Sebastian Roth

@hancocza I totally forgot to post new information in the forum as well. We were talking about this in chat and I sent an issue post to the iPXE forum because it looks like iPXE cannot handle big certificate TLS handshakes. http://forum.ipxe.org/showthread.php?tid=16998

Now I found the time to look into this again. Unfortunately there was no answer in the forum or the developers mailing list yet. Suppose they are all very busy.

Digging into the source a bit more I might have found what is causing this. The packet containing the TLS handshake with the certificate is probably being fragmented because of it’s size and iPXE is not made to handle this yet. Here I read about different kinds of fragmentation. I sure know of TCP fragmentation but I have not heard of TLS fragmentation yet. So I am trying to figure this out but it will probably take a bit more time.

hancocza

@Sebastian-Roth Thanks for checking it out. In the meantime, I ended up migrating the database to another build, and then instead of installing fog with https, I stuck with a normal install, and then reconfigured the apache2 config to redirect to https and use my certificates. Everything is working correctly now, but eventually it’d be nice to run it all in https again.

Sebastian Roth

@hancocza Unfortunately we still don’t have an answer from the (main) iPXE developers yet. From what I read between the lines the issue with certificates larger than 4096 KB is kind of known but might need a major rewrite of the code. I will keep an eye on this as I think it’s important for us and iPXE to get this fixed at some point. But I guess it won’t be any time soon. I’d like to work on this myself but have way too little time right now to dig into this part of the iPXE code way more than I have already. Will keep you posted here.

Sebastian Roth

@hancocza Just a quick update on this. I have been following the iPXE developer mailing list over the last weeks and it’s very quiet. Well there are questions coming in as well as pull requests but the developers seem to be too busy with other projects and don’t get to even review those pull requests. Probably just the current state of affairs and I guess we will see more work on iPXE at some point again. But right now I don’t see any chance of getting help from them to fix this.

hancocza

@Sebastian-Roth No problem. Like I said I’ve got it working without the certificate for now so no big deal. Thanks for being on top of it!

John Sartoris

@hancocza @Sebastian-Roth

I’ve been taking advantage of the current quarantine situation to do some needed updates to our fog servers, and found myself in the same setup. After a few other things I’ve been trying to enable HTTPS with fog using a godaddy wildcard SSL cert.

The comment about 4kb max cert is what got me to my final solution. The cert file I was using was a copy from our other apache servers and was 8kb. Clearly this is larger than 4kb. What found was that my cert had been combined with the gd_bundle-g2-g1.crt as per instructions for other hardware setups.

Turns out this isn’t required for the setup here with Apache, Fog, and iPXE. I stripped my cert file back to the original from the download zip and configured apache to run with the following lines. This shrank my cert under 4kb and allowed iPXE to work.

    SSLProtocol all -SSLv3 -SSLv2
    SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA2
    SSLHonorCipherOrder On
    SSLCertificateFile "/etc/gdssl/wildcard.solo.crt"
    SSLCertificateKeyFile "/etc/gdssl/cert_key_nopass.pem"
    SSLCACertificateFile "/etc/gdssl/gdig2.crt.pem"

Sebastian Roth

@John-Sartoris said in Not able to TFTP boot. Invalid Argument Error:

I stripped my cert file back to the original from the download zip

Wow, nice catch. I didn’t expect this to be possible because in my tests I used the official GoDaddy HTTPS URL and clearly that is using the larger certificate. Thanks for posting this!

Not able to TFTP boot. Invalid Argument Error

242

12.0k

17.3k

155.2k