TFTP Open Timeout on New Fog Install

[Sebastian Roth]

@fhrivers Please try the following: iptables -L -n -v | grep "dpt:69"

Now you see the first number in that line. That’s how often this rules has been used so far. In the output you posted this was 26 times and that means the initial TFTP connection is going through. But that is not enough. Find the output of a network packet capture below. You see first DHCP handshake (four packets) and then TFTP:

19:42:57.319383 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 08:00:27:ab:0f:cc, length 548
19:42:58.327617 IP 192.168.2.7.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 300
19:42:59.340794 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 08:00:27:ab:0f:cc, length 548
19:42:59.355690 IP 192.168.2.7.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 300
19:42:59.358575 IP 192.168.2.10.2070 > 192.168.2.7.69:  31 RRQ "undionly.kkpxe" octet tsize 0
19:42:59.378177 IP 192.168.2.7.32788 > 192.168.2.10.2070: UDP, length 14
19:42:59.378538 IP 192.168.2.10.2070 > 192.168.2.7.32788: UDP, length 17
19:42:59.379556 IP 192.168.2.10.2071 > 192.168.2.7.69:  36 RRQ "undionly.kkpxe" octet blksize 1456
19:42:59.381209 IP 192.168.2.7.60001 > 192.168.2.10.2071: UDP, length 15
19:42:59.381310 IP 192.168.2.10.2071 > 192.168.2.7.60001: UDP, length 4
19:42:59.381392 IP 192.168.2.7.60001 > 192.168.2.10.2071: UDP, length 1460
19:42:59.381717 IP 192.168.2.10.2071 > 192.168.2.7.60001: UDP, length 4
19:42:59.382154 IP 192.168.2.7.60001 > 192.168.2.10.2071: UDP, length 1460
19:42:59.382273 IP 192.168.2.10.2071 > 192.168.2.7.60001: UDP, length 4
...

While the first TFTP packet goes to UDP port 69 for the actual transfer of the file random high ports are being used. This is where your clients timeout I am fairly sure!

For now I would suggest you disable the firewall on your FOG server to see if I am on the right track. If that works then you might start reading more about Linux firewalling and how to enable it the way to still be able to use FOG. Just a hint on that: FOG uses NFS and FTP beside TFTP, which all use random ports. Therefore we usually tend to leave the firewall disabled anyway.

As well you might want to take a look at SELinux, as it can cause issues as well: https://linuxize.com/post/how-to-disable-selinux-on-centos-7/

[fhrivers]

@Sebastian-Roth Same error in Windows tftp test with firewall disabled. Its not an access denied so I’m fairly confident its not a firewall issue. I’m getting this on a VM and physical hardware install of CentOS 7.

Very strange. I even rebooted after the change for good measure.

[george1421]

@fhrivers Is your fog server, dhcp server, and pxe boot client all on the same subnet? If so lets grab a pcap of that pxe boot process. There is something going sideways here that we don’t expect.

https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

Upload the pcap to a google drive or dropbox and share the link as public. Post the link here and we will take a look at it. I recommend doing it this way because then YOU have control of the file’s existance after the debugging session is done.

Also if you install the tftp client feature in your windows computer, can you use the tftp get command to download the undionly.kpxe boot file from your FOG server? You may need to temporarily disable the windows firewall for the tftp client command to work correctly.

[fhrivers]

@george1421 Edit: Misread the question.

Everything is on the same subnet. We’re on a .23 subnet. In fact all the devices I’m using for testing are plugged into the same switch.

I’ll work on getting the info you need.

[george1421]

@fhrivers said in TFTP Open Timeout on New Fog Install:

@george1421 Fog is not handling DHCP, our corporate router does that. I’ll look at providing you that info.

As long as they are all on the same vlan (subnet) then we can get an accurate picture of what the target computer is being told.

[Sebastian Roth]

@fhrivers said in TFTP Open Timeout on New Fog Install:

Very strange. I even rebooted after the change for good measure.

Which change did you do exactly?? Disabled the firewall as suggested? I am not talking about the Windows firewall here!

Make sure firewall rules are gone after the disabled:

iptables -L -n -v
Chain INPUT (policy ACCEPT 70204 packets, 115M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 22073 packets, 26M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 64101 packets, 8327K bytes)
 pkts bytes target     prot opt in     out     source               destination         

All three default chains are empty and default policy set to ACCEPT.

[fhrivers]

@Sebastian-Roth I disabled firewalld in CentOS.

[fhrivers]

@Sebastian-Roth Here’s the output of my iptables:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

[fhrivers]

@george1421 Where is the output.pcap file saved?

[george1421]

@fhrivers It would be saved in the directory where you ran the tcpdump command.

[fhrivers]

After much frustration, I set up another Fog installation on Ubuntu that is working much better. I still get a “TFTP open timeout” but at least I can transfer TFTP in both Windows and on the Fog server.

So there’s something wrong with my CentOS install. I reinstalled it twice and followed the Wiki instructions to the T.

[fhrivers]

Okay, I managed to upload a PCAP from Wireshark:

Wireshark Capture

[george1421]

@fhrivers Looking at it now.

[george1421]

@fhrivers Look at the FOG forum chat bubble (upper right hand corner of the browser window) for additional information.

I’m almost suspecting its your dhcp server at fault here. I want you to test the following on a windows computer connected to the same subnet as the pxe booting computer.

1. Install the tftp client feature into windows 7 or windows 10 computer.
2. Temporary disable the windows firewall
3. With the tftp client feature installed key in the following into a windows command prompt tftp 172.16.10.21 GET undionly.kpxe . I’m interested in seeing if the file is downloaded. You can delete the file after it downloads, I’m only testing the process.

Let me know the results.

[fhrivers]

@george1421 I tested that earlier. It works.

[george1421]

@fhrivers said in TFTP Open Timeout on New Fog Install:

@george1421 I tested that earlier. It works.

Ah, sorry I missed that in the thread. Since tftp is working from the fog server to the windows computer, I’m going to suggest that you disable the pxe boot information on your meraki dhcp server and switch over to using dnsmasq on your FOG server. In the configuration I’m going to give you dnsmasq will only provide the pxe boot information to the client all other dhcp information will come from your main dhcp server.

I have a tutorial here on how to install dnsmasq. Use my configuration file from the tutorial just be sure to update the tag with the IP address of your FOG server. If you have clients on a different subnet then you will need to update your router’s dhcp-relay service. But I’m not seeing that is the case from your pcap file.

https://forums.fogproject.org/topic/12796/installing-dnsmasq-on-your-fog-server

[george1421]

The OP was able to get things running using dnsmasq. It appears that the meraki dhcp server doesn’t provide enough information to the pxe booting computer. The advantage of dnsmasq (as well as simply working) it will dynamically provide the correct pxe boot loader based on the pxe booting client. That is something the meraki dhcp server isn’t capable of providing.

[fhrivers]

Much thanks, George and others for the help. As well as saving my company thousands of dollars!