TFTP Open Timeout on New Fog Install

fhrivers

Prior to this role I installed Fog 0.32 and didn’t have as much issues as I’m having with this latest version. I’m unable to connect to the FOG tftp when booting from the network interface. I get the error “TFTP Open Timeout” on my Lenovo X1 Carbon and my VMWare Workstation VM pulls a DHCP address and just fails to the BIOS.

I admit that I had the wrong IP address in the option 66, noticed it right away and updated the IP address but machines still fail with the same error. Option 67 is set to undionly.kpxe like it’s supposed to. Both clients still failing.

My DHCP server is a Meraki MX100.

george1421

@fhrivers said in TFTP Open Timeout on New Fog Install:

@george1421 I tested that earlier. It works.

Ah, sorry I missed that in the thread. Since tftp is working from the fog server to the windows computer, I’m going to suggest that you disable the pxe boot information on your meraki dhcp server and switch over to using dnsmasq on your FOG server. In the configuration I’m going to give you dnsmasq will only provide the pxe boot information to the client all other dhcp information will come from your main dhcp server.

I have a tutorial here on how to install dnsmasq. Use my configuration file from the tutorial just be sure to update the tag with the IP address of your FOG server. If you have clients on a different subnet then you will need to update your router’s dhcp-relay service. But I’m not seeing that is the case from your pcap file.

https://forums.fogproject.org/topic/12796/installing-dnsmasq-on-your-fog-server

george1421

@fhrivers So pxe booting worked OK with 0.3x version of fog and you only updated (installed) 1.5.5 and have pxe booting issues? Was there any gap in time between you using fog 0.3x and 1.5.5?

Sebastian Roth

@fhrivers said in TFTP Open Timeout on New Fog Install:

Option 67 is set to undionly.kpxe like it’s supposed to

Not for UEFI booting machines though. If your machines are UEFI booting then you need to use ipxe.efi. Details on this see here: https://wiki.fogproject.org/wiki/index.php?title=BIOS_and_UEFI_Co-Existence

I guess we need more details to be able to help:

Which Linux OS do you use? CentOS? Debian? Ubuntu? Did you disable SELinux and the firewall?
Is TFTP running? Command to check: netstat -antup | grep ":69" or ss -antul | grep ":69" (if netstat is not installed)
Do you have other machines (beside Lenovo X1 Carbon and my VMWare Workstation VM which can both be tricky) that do properly PXE boot on the new FOG server?
Take a picture of the actual TFTP error on screen an post here!
Have you read the troubleshooting guide in our wiki? https://wiki.fogproject.org/wiki/index.php?title=Tftp_timeout… (please follow the tests to see if you can manually download the iPXE binary file)

fhrivers

@george1421 3 years and one company! LOL. Also did that in a VMWare environment. Lots of differences this time!

fhrivers

@Sebastian-Roth Netstat returns this:

udp 0 0 0.0.0.0:69 0.0.0.0:* 3332/xinetd

fhrivers

@Sebastian-Roth said in TFTP Open Timeout on New Fog Install:

ipxe.efi

Error message attached.

PXE error.jpg

fhrivers

Okay, sorry about the spam as I’m troubleshooting in parallel with providing you with more information. I’m running Fog in a CentOS VM running in Virtualbox which is running on a Windows 10 host. So I may be running into Windows firewall issues. I cannot ping external machines on the network from CentOS.

Sebastian Roth

@fhrivers Take a look at this wiki article:
https://wiki.fogproject.org/wiki/index.php?title=CentOS_7#Continue_pre-config

As well I am wondering how you setup the networking of the VirtualBox VM. Best if you can take a picture of the network settings in VirtualBox and post here.

fhrivers

@Sebastian-Roth

VirtualBox is running on a Windows 10 PC and is setup in Bridge mode. I had some networking issues due to an aggressive web filter on our network, but other than that, this has been functioning normally.

Please disregard the fact that I couldn’t ping anything from the server. I can’t ping them from other machines from the host either. So I’m not pursuing that path right now.

I did view the TFTP troubleshooting page and I checked everything there with no success.

fhrivers

Just for good measure, I tore out the VirtualBox solution and stood FOG up on an old desktop. Same error.

I even ran tftp -i <ip> get udionly.kpxe

connect request failed

Same problem with legacy BIOS and UEFI. I can ping the FOG server from my client machine, but FOG can’t ping client.

Sebastian Roth

@fhrivers Run iptables -L -n -v to see if the local firewall on the Linux system is enabled. Take a picture and post here.

fhrivers

@Sebastian-Roth Attached is output.

CentOS FW output.pdf

Sebastian Roth

@fhrivers Please try the following: iptables -L -n -v | grep "dpt:69"

Now you see the first number in that line. That’s how often this rules has been used so far. In the output you posted this was 26 times and that means the initial TFTP connection is going through. But that is not enough. Find the output of a network packet capture below. You see first DHCP handshake (four packets) and then TFTP:

19:42:57.319383 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 08:00:27:ab:0f:cc, length 548
19:42:58.327617 IP 192.168.2.7.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 300
19:42:59.340794 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 08:00:27:ab:0f:cc, length 548
19:42:59.355690 IP 192.168.2.7.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 300
19:42:59.358575 IP 192.168.2.10.2070 > 192.168.2.7.69:  31 RRQ "undionly.kkpxe" octet tsize 0
19:42:59.378177 IP 192.168.2.7.32788 > 192.168.2.10.2070: UDP, length 14
19:42:59.378538 IP 192.168.2.10.2070 > 192.168.2.7.32788: UDP, length 17
19:42:59.379556 IP 192.168.2.10.2071 > 192.168.2.7.69:  36 RRQ "undionly.kkpxe" octet blksize 1456
19:42:59.381209 IP 192.168.2.7.60001 > 192.168.2.10.2071: UDP, length 15
19:42:59.381310 IP 192.168.2.10.2071 > 192.168.2.7.60001: UDP, length 4
19:42:59.381392 IP 192.168.2.7.60001 > 192.168.2.10.2071: UDP, length 1460
19:42:59.381717 IP 192.168.2.10.2071 > 192.168.2.7.60001: UDP, length 4
19:42:59.382154 IP 192.168.2.7.60001 > 192.168.2.10.2071: UDP, length 1460
19:42:59.382273 IP 192.168.2.10.2071 > 192.168.2.7.60001: UDP, length 4
...

While the first TFTP packet goes to UDP port 69 for the actual transfer of the file random high ports are being used. This is where your clients timeout I am fairly sure!

For now I would suggest you disable the firewall on your FOG server to see if I am on the right track. If that works then you might start reading more about Linux firewalling and how to enable it the way to still be able to use FOG. Just a hint on that: FOG uses NFS and FTP beside TFTP, which all use random ports. Therefore we usually tend to leave the firewall disabled anyway.

As well you might want to take a look at SELinux, as it can cause issues as well: https://linuxize.com/post/how-to-disable-selinux-on-centos-7/

fhrivers

@Sebastian-Roth Same error in Windows tftp test with firewall disabled. Its not an access denied so I’m fairly confident its not a firewall issue. I’m getting this on a VM and physical hardware install of CentOS 7.

Very strange. I even rebooted after the change for good measure.

george1421

@fhrivers Is your fog server, dhcp server, and pxe boot client all on the same subnet? If so lets grab a pcap of that pxe boot process. There is something going sideways here that we don’t expect.

https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

Upload the pcap to a google drive or dropbox and share the link as public. Post the link here and we will take a look at it. I recommend doing it this way because then YOU have control of the file’s existance after the debugging session is done.

Also if you install the tftp client feature in your windows computer, can you use the tftp get command to download the undionly.kpxe boot file from your FOG server? You may need to temporarily disable the windows firewall for the tftp client command to work correctly.

fhrivers

@george1421 Edit: Misread the question.

Everything is on the same subnet. We’re on a .23 subnet. In fact all the devices I’m using for testing are plugged into the same switch.

I’ll work on getting the info you need.

george1421

@fhrivers said in TFTP Open Timeout on New Fog Install:

@george1421 Fog is not handling DHCP, our corporate router does that. I’ll look at providing you that info.

As long as they are all on the same vlan (subnet) then we can get an accurate picture of what the target computer is being told.

Sebastian Roth

@fhrivers said in TFTP Open Timeout on New Fog Install:

Very strange. I even rebooted after the change for good measure.

Which change did you do exactly?? Disabled the firewall as suggested? I am not talking about the Windows firewall here!

Make sure firewall rules are gone after the disabled:

iptables -L -n -v
Chain INPUT (policy ACCEPT 70204 packets, 115M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 22073 packets, 26M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 64101 packets, 8327K bytes)
 pkts bytes target     prot opt in     out     source               destination

All three default chains are empty and default policy set to ACCEPT.

fhrivers

@Sebastian-Roth I disabled firewalld in CentOS.

fhrivers

@Sebastian-Roth Here’s the output of my iptables:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

TFTP Open Timeout on New Fog Install

135

12.1k

17.3k

155.3k