TFTP Open Timeout on New Fog Install

[fhrivers]

Okay, sorry about the spam as I’m troubleshooting in parallel with providing you with more information. I’m running Fog in a CentOS VM running in Virtualbox which is running on a Windows 10 host. So I may be running into Windows firewall issues. I cannot ping external machines on the network from CentOS.

[Sebastian Roth]

@fhrivers Take a look at this wiki article:
https://wiki.fogproject.org/wiki/index.php?title=CentOS_7#Continue_pre-config

As well I am wondering how you setup the networking of the VirtualBox VM. Best if you can take a picture of the network settings in VirtualBox and post here.

[fhrivers]

@Sebastian-Roth

VirtualBox is running on a Windows 10 PC and is setup in Bridge mode. I had some networking issues due to an aggressive web filter on our network, but other than that, this has been functioning normally.

Please disregard the fact that I couldn’t ping anything from the server. I can’t ping them from other machines from the host either. So I’m not pursuing that path right now.

I did view the TFTP troubleshooting page and I checked everything there with no success.

[fhrivers]

Just for good measure, I tore out the VirtualBox solution and stood FOG up on an old desktop. Same error.

I even ran tftp -i <ip> get udionly.kpxe

connect request failed

Same problem with legacy BIOS and UEFI. I can ping the FOG server from my client machine, but FOG can’t ping client.

[Sebastian Roth]

@fhrivers Run iptables -L -n -v to see if the local firewall on the Linux system is enabled. Take a picture and post here.

[fhrivers]

@Sebastian-Roth Attached is output.

CentOS FW output.pdf

[Sebastian Roth]

@fhrivers Please try the following: iptables -L -n -v | grep "dpt:69"

Now you see the first number in that line. That’s how often this rules has been used so far. In the output you posted this was 26 times and that means the initial TFTP connection is going through. But that is not enough. Find the output of a network packet capture below. You see first DHCP handshake (four packets) and then TFTP:

19:42:57.319383 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 08:00:27:ab:0f:cc, length 548
19:42:58.327617 IP 192.168.2.7.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 300
19:42:59.340794 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 08:00:27:ab:0f:cc, length 548
19:42:59.355690 IP 192.168.2.7.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 300
19:42:59.358575 IP 192.168.2.10.2070 > 192.168.2.7.69:  31 RRQ "undionly.kkpxe" octet tsize 0
19:42:59.378177 IP 192.168.2.7.32788 > 192.168.2.10.2070: UDP, length 14
19:42:59.378538 IP 192.168.2.10.2070 > 192.168.2.7.32788: UDP, length 17
19:42:59.379556 IP 192.168.2.10.2071 > 192.168.2.7.69:  36 RRQ "undionly.kkpxe" octet blksize 1456
19:42:59.381209 IP 192.168.2.7.60001 > 192.168.2.10.2071: UDP, length 15
19:42:59.381310 IP 192.168.2.10.2071 > 192.168.2.7.60001: UDP, length 4
19:42:59.381392 IP 192.168.2.7.60001 > 192.168.2.10.2071: UDP, length 1460
19:42:59.381717 IP 192.168.2.10.2071 > 192.168.2.7.60001: UDP, length 4
19:42:59.382154 IP 192.168.2.7.60001 > 192.168.2.10.2071: UDP, length 1460
19:42:59.382273 IP 192.168.2.10.2071 > 192.168.2.7.60001: UDP, length 4
...

While the first TFTP packet goes to UDP port 69 for the actual transfer of the file random high ports are being used. This is where your clients timeout I am fairly sure!

For now I would suggest you disable the firewall on your FOG server to see if I am on the right track. If that works then you might start reading more about Linux firewalling and how to enable it the way to still be able to use FOG. Just a hint on that: FOG uses NFS and FTP beside TFTP, which all use random ports. Therefore we usually tend to leave the firewall disabled anyway.

As well you might want to take a look at SELinux, as it can cause issues as well: https://linuxize.com/post/how-to-disable-selinux-on-centos-7/

[fhrivers]

@Sebastian-Roth Same error in Windows tftp test with firewall disabled. Its not an access denied so I’m fairly confident its not a firewall issue. I’m getting this on a VM and physical hardware install of CentOS 7.

Very strange. I even rebooted after the change for good measure.

[george1421]

@fhrivers Is your fog server, dhcp server, and pxe boot client all on the same subnet? If so lets grab a pcap of that pxe boot process. There is something going sideways here that we don’t expect.

https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

Upload the pcap to a google drive or dropbox and share the link as public. Post the link here and we will take a look at it. I recommend doing it this way because then YOU have control of the file’s existance after the debugging session is done.

Also if you install the tftp client feature in your windows computer, can you use the tftp get command to download the undionly.kpxe boot file from your FOG server? You may need to temporarily disable the windows firewall for the tftp client command to work correctly.

[fhrivers]

@george1421 Edit: Misread the question.

Everything is on the same subnet. We’re on a .23 subnet. In fact all the devices I’m using for testing are plugged into the same switch.

I’ll work on getting the info you need.

[george1421]

@fhrivers said in TFTP Open Timeout on New Fog Install:

@george1421 Fog is not handling DHCP, our corporate router does that. I’ll look at providing you that info.

As long as they are all on the same vlan (subnet) then we can get an accurate picture of what the target computer is being told.

[Sebastian Roth]

@fhrivers said in TFTP Open Timeout on New Fog Install:

Very strange. I even rebooted after the change for good measure.

Which change did you do exactly?? Disabled the firewall as suggested? I am not talking about the Windows firewall here!

Make sure firewall rules are gone after the disabled:

iptables -L -n -v
Chain INPUT (policy ACCEPT 70204 packets, 115M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 22073 packets, 26M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 64101 packets, 8327K bytes)
 pkts bytes target     prot opt in     out     source               destination         

All three default chains are empty and default policy set to ACCEPT.

[fhrivers]

@Sebastian-Roth I disabled firewalld in CentOS.

[fhrivers]

@Sebastian-Roth Here’s the output of my iptables:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

[fhrivers]

@george1421 Where is the output.pcap file saved?

[george1421]

@fhrivers It would be saved in the directory where you ran the tcpdump command.

[fhrivers]

After much frustration, I set up another Fog installation on Ubuntu that is working much better. I still get a “TFTP open timeout” but at least I can transfer TFTP in both Windows and on the Fog server.

So there’s something wrong with my CentOS install. I reinstalled it twice and followed the Wiki instructions to the T.

[fhrivers]

Okay, I managed to upload a PCAP from Wireshark:

Wireshark Capture

[george1421]

@fhrivers Looking at it now.

[george1421]

@fhrivers Look at the FOG forum chat bubble (upper right hand corner of the browser window) for additional information.

I’m almost suspecting its your dhcp server at fault here. I want you to test the following on a windows computer connected to the same subnet as the pxe booting computer.

1. Install the tftp client feature into windows 7 or windows 10 computer.
2. Temporary disable the windows firewall
3. With the tftp client feature installed key in the following into a windows command prompt tftp 172.16.10.21 GET undionly.kpxe . I’m interested in seeing if the file is downloaded. You can delete the file after it downloads, I’m only testing the process.

Let me know the results.