TFTP Timeout

Tauric

Hi guys, I recently decided to move from WDS to Fog and have began setting it up today but have ran into some issues with TFTP-HPA

I have a VMWare workstation VM running the latest release of ubuntu (freshly installed today), any TFTP get requests (either manually or from PXE) seem to be timing out.

Running tftp from windows shows

tftp 192.168.0.26 get ipxe.efi
 Connect request failed

with the TCPDump showing:

    192.168.0.87.53571 > pcNameHere.tftp: [udp sum ok] TFTP, length 20, RRQ "ipxe.efi" netascii
    
    16:17:05.206384 IP (tos 0x0, ttl 128, id 3541, offset 0, flags [none], proto UDP (17), length 48)
    
        192.168.0.87.53571 > pcNameHere.tftp: [udp sum ok] TFTP, length 20, RRQ "ipxe.efi" netascii
    
    16:17:07.212387 IP (tos 0x0, ttl 128, id 3542, offset 0, flags [none], proto UDP (17), length 48)
    
        192.168.0.87.53571 > pcNameHere.tftp: [udp sum ok] TFTP, length 20, RRQ "ipxe.efi" netascii
    
    16:17:11.219890 IP (tos 0x0, ttl 128, id 3543, offset 0, flags [none], proto UDP (17), length 48)
    
        192.168.0.87.53571 > pcNameHere.tftp: [udp sum ok] TFTP, length 20, RRQ "ipxe.efi" netascii
    
    16:17:19.225284 IP (tos 0x0, ttl 128, id 3549, offset 0, flags [none], proto UDP (17), length 48)
    
        192.168.0.87.53571 > pcNameHere.tftp: [udp sum ok] TFTP, length 20, RRQ "ipxe.efi" netascii
    
    16:17:27.229313 IP (tos 0x0, ttl 128, id 3550, offset 0, flags [none], proto UDP (17), length 48)
    
        192.168.0.87.53571 > pcNameHere.tftp: [udp sum ok] TFTP, length 20, RRQ "ipxe.efi" netascii
    
    16:17:35.237293 IP (tos 0x0, ttl 128, id 3551, offset 0, flags [none], proto UDP (17), length 48)
    
        192.168.0.87.53571 > pcNameHere.tftp: [udp sum ok] TFTP, length 20, RRQ "ipxe.efi" netascii
    
    16:17:43.239364 IP (tos 0x0, ttl 128, id 3552, offset 0, flags [none], proto UDP (17), length 48)
    
        192.168.0.87.53571 > pcNameHere.tftp: [udp sum ok] TFTP, length 20, RRQ "ipxe.efi" netascii
    
    16:17:51.245345 IP (tos 0x0, ttl 128, id 3553, offset 0, flags [none], proto UDP (17), length 51)
    
        192.168.0.87.53571 > pcNameHere.tftp: [udp sum ok] TFTP, length 23, ERROR EUNDEF "timeout on receive"

The tftpboot permissions are:

/var/log# ls -l /tftpboot
total 6164
drwxr-xr-x 4 fogproject root    4096 Oct 27 15:14 10secdelay
drwxr-xr-x 2 fogproject root    4096 Oct 27 15:14 arm64-efi
-rw-r--r-- 1 root       root     457 Oct 27 15:14 default.ipxe
drwxr-xr-x 2 fogproject root    4096 Oct 27 15:14 i386-efi
-rw-r-xr-x 1 fogproject root  268288 Oct 27 15:14 intel.efi
-rw-r-xr-x 1 fogproject root  104229 Oct 27 15:14 intel.kkpxe
-rw-r-xr-x 1 fogproject root  104277 Oct 27 15:14 intel.kpxe
-rw-r-xr-x 1 fogproject root  104233 Oct 27 15:14 intel.pxe
-rw-r-xr-x 1 fogproject root 1093120 Oct 27 15:14 ipxe.efi
-rw-r-xr-x 1 fogproject root  907264 Oct 27 15:14 ipxe.iso
-rw-r-xr-x 1 fogproject root  371566 Oct 27 15:14 ipxe.kkpxe
-rw-r-xr-x 1 fogproject root  371614 Oct 27 15:14 ipxe.kpxe
-rw-r-xr-x 1 fogproject root  370994 Oct 27 15:14 ipxe.krn
-rw-r-xr-x 1 fogproject root  370994 Oct 27 15:14 ipxe.lkrn
-rw-r-xr-x 1 fogproject root  371834 Oct 27 15:14 ipxe.pxe
-rw-r-xr-x 1 fogproject root  409600 Oct 27 15:14 ipxe.usb
-rw-r-xr-x 1 fogproject root   26140 Oct 27 15:14 memdisk
-rw-r-xr-x 1 fogproject root  299008 Oct 27 15:14 ncm--ecm--axge.efi
-rw-r-xr-x 1 fogproject root  266240 Oct 27 15:14 realtek.efi
-rw-r-xr-x 1 fogproject root  104968 Oct 27 15:14 realtek.kkpxe
-rw-r-xr-x 1 fogproject root  105016 Oct 27 15:14 realtek.kpxe
-rw-r-xr-x 1 fogproject root  104987 Oct 27 15:14 realtek.pxe
-rw-r-xr-x 1 fogproject root  266240 Oct 27 15:14 snp.efi
-rw-r-xr-x 1 fogproject root  266752 Oct 27 15:14 snponly.efi
-rw-r-xr-x 1 fogproject root  103589 Oct 27 15:14 undionly.kkpxe
-rw-r-xr-x 1 fogproject root  103637 Oct 27 15:14 undionly.kpxe
-rw-r-xr-x 1 fogproject root  103666 Oct 27 15:14 undionly.pxe

UFW is disabled (fresh install so it shouldn’t be anyway)

/var/log# sudo ufw status
Status: inactive

tftp-hpa config:

/var/log# cat /etc/default/tftpd-hpa 
# /etc/default/tftpd-hpa
# FOG Modified version
TFTP_USERNAME="root"
TFTP_DIRECTORY="/tftpboot"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="-s"

tftpd-hpa status:

/var/log# sudo systemctl status tftpd-hpa
● tftpd-hpa.service - LSB: HPA's tftp server
     Loaded: loaded (/etc/init.d/tftpd-hpa; generated)
     Active: active (running) since Fri 2023-10-27 15:52:46 BST; 29min ago
       Docs: man:systemd-sysv-generator(8)
    Process: 47694 ExecStart=/etc/init.d/tftpd-hpa start (code=exited, status=0/SUCCESS)
      Tasks: 1 (limit: 4556)
     Memory: 440.0K
        CPU: 72ms
     CGroup: /system.slice/tftpd-hpa.service
             └─47702 /usr/sbin/in.tftpd --listen --user root --address :69 -s /tftpboot

Oct 27 15:52:46 fbs-virtual-machine systemd[1]: Starting LSB: HPA's tftp server...
Oct 27 15:52:46 fbs-virtual-machine tftpd-hpa[47694]:  * Starting HPA's tftpd in.tftpd
Oct 27 15:52:46 fbs-virtual-machine tftpd-hpa[47694]:    ...done.
Oct 27 15:52:46 fbs-virtual-machine systemd[1]: Started LSB: HPA's tftp server.

Thanks for any suggestions

Edit: the VM’s are all bridged networks with their own IP’s which works fine with WDS

george1421

@Tauric ok I see a whole lot of issues here. Let me ask you did you mask out any data in the pcap?

In the ethernet header (bootp protocol) the boot-file field is blank (should be ipxe.efi). The next server points to 192.168.0.254 not 0.33) Looking at the dhcp part, dhcp option 66 (should be boot server IP) is an unpritable character. DHCP option 67 is ipxe.efi but its not terminated with an end of string character 0x00, it ends the string with 0xFF. For background both bootp and dhcp options need to be set because its up to the pxe rom writer to pick if they want to boot using bootp (older protocol) or dhcp. The issue here is with your dhcp server giving your target computers bad info.

Since you are using a SOHO router, we see them not exactly place nice with pxe booting.

My recommendation is if you can’t fix your dhcp server easily then forgo using it and install dnsmasq on your fog server. It will take about 10 minutes as well as support dynamic pxe booting (bios/uefi). DNSMASQ in this configuration will not issue IP address, but only pxe boot into. https://forums.fogproject.org/topic/12796/installing-dnsmasq-on-your-fog-server?_=1698421239631

george1421

@Tauric On the windows box, make sure you disable the firewall since tftp uses 2 network ports like ftp does, if you are trying to make a comparative test.

Since you seem confident with tcpdump. Lets follow this tutorial to get a pcap from the FOG server. This will show us the dhcp process as well as the tftp process at the end. It should give us a good picture of what is going on. Capture the pcap and upload it to a file share site and I will take a look at it. https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue?_=1698421239623 I’ll need the complete pcap since the screen shots don’t show the complete details and there are a many exceptions to list.

Tauric

This post is deleted!

Tauric

@george1421

Hi George,

Thanks a lot for the quick response here, it appears the tftp GET works with the windows firewall disabled:

tftp -i 192.168.0.26 GET ipxe.efi                           
Transfer successful: 1093120 bytes in 1 second(s), 1093120 bytes/s

So that obviously points to something blocking the ephemeral port of the boxes that are pxe booting, however I’m unsure as to where this would be, the router is a Draytek and has the firewall disabled and UFW is off, unless theres another firewall i’m not aware of in ubuntu?

In the attached PCAP the first two requests (as well as one about halfway down) are informs from IP phones, there are 6 read requests before the PXE process fails.

Draytek (router, dhcp and dns): 0.254
PXE’ing box: 0.33
Ubuntu FOG server: 0.26

PXEoutput.pcap

Thanks again

george1421

@Tauric ok I see a whole lot of issues here. Let me ask you did you mask out any data in the pcap?

In the ethernet header (bootp protocol) the boot-file field is blank (should be ipxe.efi). The next server points to 192.168.0.254 not 0.33) Looking at the dhcp part, dhcp option 66 (should be boot server IP) is an unpritable character. DHCP option 67 is ipxe.efi but its not terminated with an end of string character 0x00, it ends the string with 0xFF. For background both bootp and dhcp options need to be set because its up to the pxe rom writer to pick if they want to boot using bootp (older protocol) or dhcp. The issue here is with your dhcp server giving your target computers bad info.

Since you are using a SOHO router, we see them not exactly place nice with pxe booting.

My recommendation is if you can’t fix your dhcp server easily then forgo using it and install dnsmasq on your fog server. It will take about 10 minutes as well as support dynamic pxe booting (bios/uefi). DNSMASQ in this configuration will not issue IP address, but only pxe boot into. https://forums.fogproject.org/topic/12796/installing-dnsmasq-on-your-fog-server?_=1698421239631

Tauric

@george1421 I sent the pcap as is, aside from renaming it, I moved it from the ubuntu machine to an smb share on my windows box before uploading it here, I did notice the unprintable characters but I just assumed that was some issue with how the pcap was captured, I checked in the draytek and the options look fine, 66 is set as an address and the value is 192.168.0.26, not sure what you mean about the trailing character on ipxe.efi for 67.

Cheers for the advice on this, ill probably just go with the dnsmasq as suggested

Thanks for the help
-Tauric

george1421

@Tauric The question about editing the pcap, I’ve seen some people mask info in the pcap thinking about privacy, but that just adds confusion, like the unprintable characters. I thought the unprintable characters were the results of hand editing the pcap file.

The advantage of going the dnsmasq route on the fog server is if the fog server isn’t running you have nothing issues pxe boot into. If you go the dnsmasq route remote the pxe boot information in your router so it doesn’t confuse things when the fog server is offline

Tauric

@george1421 That makes sense, i’ll mark the thread as solved tomorrow if that works, the process does look pretty simple

I’m unsure as to why the Draytek router is sending the DHCP options like this, but honestly it is a bit old and they aren’t the most reliable in general haha

Tauric

@george1421 Just got around to testing dnsmasq today, It worked flawlessly (and took less than 10 minutes lol). Thanks for the help here

george1421

@Tauric said in TFTP Timeout:

(and took less than 10 minutes lol)

Well I was taking into account for slow speeds between keyboard and chair…
Glad you have it worked out. DNSMASQ should work flawlessly in your environment.

TFTP Timeout

204

12.0k

17.3k

155.1k