Not able to TFTP boot. Invalid Argument Error

[Sebastian Roth]

@hancocza Oh man/woman, I have overlooked the most obvious problem with this. iPXE is connecting to the IP address instead of hostname and therefore cannot match the names in the certificate. I just had a quick look at the manual pages and seems like iPXE can handle alternative subjects, so I was wrong in my last post.

Edit /tftpboot/default.ipxe on your FOG server and make it use the DNS name instead of the IP.

[hancocza]

@Sebastian-Roth the default.ipxe file already specifies the fqdn of fogserver.gvsu.edu. on 1.5.0, I would just change it to the IP and change https to http and not recompile, and it all worked fine. Since we’ve been doing this, I’ve left it as https://fogserver.gvsu.edu/…

If it could still be an issue of the two separate certs (ca and server) in the buildipxe.sh, I’m not sure what to use as the ca one. Go Daddy uses an intermediate cert instead of a CA, but that was the chain one that we weren’t sure if it’d work.

[Sebastian Roth]

@hancocza Please check to see if you have also set the FQDN in FOG configuration -> FOG settings -> Web Server -> WEB HOST! Writing my last messages in the train where I don’t have good access to the code I forgot about this.

From the information you posted I would expect you’d need to use the GoDaddy Secure Server Certificate (Intermediate Certificate) - G2 as TRUST. This (or more precisely its private key) should be exactly the one that was used to sign your certificate.

[hancocza]

@Sebastian-Roth said in Not able to TFTP boot. Invalid Argument Error:

@hancocza Please check to see if you have also set the FQDN in FOG configuration -> FOG settings -> Web Server -> WEB HOST! Writing my last messages in the train where I don’t have good access to the code I forgot about this.

From the information you posted I would expect you’d need to use the GoDaddy Secure Server Certificate (Intermediate Certificate) - G2 as TRUST. This (or more precisely its private key) should be exactly the one that was used to sign your certificate.

I had the IP set in the WEB HOST field. I switched that now. Also updated the TRUST parameter in buildipxe.sh to have that cert. I’ll test tomorrow and see what happens!

[hancocza]

@Sebastian-Roth Just tested, still a no go. Trust line looks like this with the intermediate cert:

BUILDOPTS=“CERT=/home/fogserver/Documents/csims.clas.gvsu.edu/fogcert.pem TRUST=/home/fogserver/Documents/csims.clas.gvsu.edu/GoDaddyCA.pem”

[Sebastian Roth]

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

GoDaddyCA.pem

Sure this is the same one that I pointed out? Please run the following commands to check:

cd /tmp/
wget https://certs.godaddy.com/repository/gdig2.crt.pem
sha256sum gdig2.crt.pem /home/fogserver/Documents/csims.clas.gvsu.edu/GoDaddyCA.pem
...

See if you get the same hash for both files.

You might need to use different ones for iPXE versus the ones used for Apache. Apache needs the full trust chain I reckon. You have some major root CA certificates installed on your system already and point Apache to some chain certificates (possibly two intermediate ones) so the chain is fully trusted. For iPXE I guess (sorry, don’t have enough time to test or look it up right now) you only need to trust the one single certificate/key that was used to issue your web server certificate and I think that is https://certs.godaddy.com/repository/gdig2.crt.pem in your case.

[hancocza]

@Sebastian-Roth It wasn’t the one you mentioned. I downloaded the correct one and put it in place, still got an invalid argument error. Do i need to append my certificate to the trust as well, so that it looks like:

TRUST=/home/fogserver/Desktop/gdig2.crt.pem,/home/fogserver/Documents/csims.clas.gvsu.edu/fogcert.crt

I was looking at this website when I came across that: https://ipxe.org/crypto

[Sebastian Roth]

@hancocza said in Not able to TFTP boot. Invalid Argument Error:

Do i need to append my certificate to the trust as well

Don’t think you have to. But reading more about how (open)ssl certificate verification works I figured that I was wrong with one of my assumptions. The intermediate certificate is not enough as it is not selfiestick signed (only root certs are). So you’d need to specify intermediate and root cert as TRUST.

[Sebastian Roth]

@hancocza Try using TRUST=gdroot-g2.crt.pem,gdig2.crt.pem

https://certs.godaddy.com/repository/gdroot-g2.crt
https://certs.godaddy.com/repository/gdig2.crt.pem

I think you need to convert the gdroot-g2.crt to PEM format. I read that iPXE can only handle PEM but not DER cert format. Convert using openssl x509 -inform DER -outform PEM -text -in gdroot-g2.crt -out gdroot-g2.crt.pem

You might want to check the whole chain using openssl as well: openssl verify -CAfile gdroot-g2.crt.pem -untrusted gdig2.crt.pem fogcert.pem

all untested…

EDIT: Posted on the iPXE developers mailinglist: http://lists.ipxe.org/pipermail/ipxe-devel/2018-December/006395.html

[hancocza]

@Sebastian-Roth Looks like i didn’t need to convert the root certificate, it was already in pem format. I ran the openssl verify command, and it returned an ‘OK’. I’ll insert that into the trust and recompile. I wont be able to test until tomorrow morning, so I’ll update you then. Thanks for all the help!

[hancocza]

@Sebastian-Roth Just tested, still the same. I ran the certstat command in the ipxe shell and it only listed the fogcert.crt certificate. Should it also have the other ones listed?

[Sebastian Roth]

@hancocza I totally forgot to post new information in the forum as well. We were talking about this in chat and I sent an issue post to the iPXE forum because it looks like iPXE cannot handle big certificate TLS handshakes. http://forum.ipxe.org/showthread.php?tid=16998

Now I found the time to look into this again. Unfortunately there was no answer in the forum or the developers mailing list yet. Suppose they are all very busy.

Digging into the source a bit more I might have found what is causing this. The packet containing the TLS handshake with the certificate is probably being fragmented because of it’s size and iPXE is not made to handle this yet. Here I read about different kinds of fragmentation. I sure know of TCP fragmentation but I have not heard of TLS fragmentation yet. So I am trying to figure this out but it will probably take a bit more time.

[hancocza]

@Sebastian-Roth Thanks for checking it out. In the meantime, I ended up migrating the database to another build, and then instead of installing fog with https, I stuck with a normal install, and then reconfigured the apache2 config to redirect to https and use my certificates. Everything is working correctly now, but eventually it’d be nice to run it all in https again.

[Sebastian Roth]

@hancocza Unfortunately we still don’t have an answer from the (main) iPXE developers yet. From what I read between the lines the issue with certificates larger than 4096 KB is kind of known but might need a major rewrite of the code. I will keep an eye on this as I think it’s important for us and iPXE to get this fixed at some point. But I guess it won’t be any time soon. I’d like to work on this myself but have way too little time right now to dig into this part of the iPXE code way more than I have already. Will keep you posted here.

[Sebastian Roth]

@hancocza Just a quick update on this. I have been following the iPXE developer mailing list over the last weeks and it’s very quiet. Well there are questions coming in as well as pull requests but the developers seem to be too busy with other projects and don’t get to even review those pull requests. Probably just the current state of affairs and I guess we will see more work on iPXE at some point again. But right now I don’t see any chance of getting help from them to fix this.

[hancocza]

@Sebastian-Roth No problem. Like I said I’ve got it working without the certificate for now so no big deal. Thanks for being on top of it!

[John Sartoris]

@hancocza @Sebastian-Roth

I’ve been taking advantage of the current quarantine situation to do some needed updates to our fog servers, and found myself in the same setup. After a few other things I’ve been trying to enable HTTPS with fog using a godaddy wildcard SSL cert.

The comment about 4kb max cert is what got me to my final solution. The cert file I was using was a copy from our other apache servers and was 8kb. Clearly this is larger than 4kb. What found was that my cert had been combined with the gd_bundle-g2-g1.crt as per instructions for other hardware setups.

Turns out this isn’t required for the setup here with Apache, Fog, and iPXE. I stripped my cert file back to the original from the download zip and configured apache to run with the following lines. This shrank my cert under 4kb and allowed iPXE to work.

    SSLProtocol all -SSLv3 -SSLv2
    SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA2
    SSLHonorCipherOrder On
    SSLCertificateFile "/etc/gdssl/wildcard.solo.crt"
    SSLCertificateKeyFile "/etc/gdssl/cert_key_nopass.pem"
    SSLCACertificateFile "/etc/gdssl/gdig2.crt.pem"

[Sebastian Roth]

@John-Sartoris said in Not able to TFTP boot. Invalid Argument Error:

I stripped my cert file back to the original from the download zip

Wow, nice catch. I didn’t expect this to be possible because in my tests I used the official GoDaddy HTTPS URL and clearly that is using the larger certificate. Thanks for posting this!

[Sebastian Roth]

@hancocza @John-Sartoris Hey, can hardly believe it but there is hope. Someone got around to implement a fix and send in a pull request to the iPXE devs: https://github.com/ipxe/ipxe/pull/116

[hancocza]

@Sebastian-Roth that’s good news! I can test it out once it’s wrapped in. I tried what John did, but didn’t have any luck. Also tried following the wiki post on HTTPs/custom certificates that you shared in the other thread, but was still getting the overlength error. I decided to just use the certificate that FOG creates on install instead of my custom one for now.