Is it possible to connect to the Fog server remotely?

[george1421]

@pjb1983 Try the same test with a packet size of 1461

If we find that the MTU is 1400 we can set the tftp block size just a bit smaller so the packets don’t fragment.

What host OS is your FOG server using?

[PJB1983]

This post is deleted!

[PJB1983]

@george1421 the max MTU size is 1404.
I’ve installed the FOG server on a Linux Ubuntu.

[Sebastian Roth]

@george1421 said in Is it possible to connect to the Fog server remotely?:

If we find that the MTU is 1400 we can set the tftp block size just a bit smaller so the packets don’t fragment.

That would get you past the TFTP part but it would fail downloading the kernel and init over HTTP then I suppose.

https://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html

[Sebastian Roth]

@PJB1983 This might be a quick win: https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/82444-fragmentation.html#enc-error

[PJB1983]

@sebastian-roth thx for your reply, but i doesn’t have a CLI on the Meraki Firewall. I have to contact support i guess.

[george1421]

@sebastian-roth I think our only option is to handle this on the tftp server size. I think the issue is packet fragmentation over UDP. I think we can set a maximum block size for the tftp server. What we need to do is set it at -64b from the MTU. I did find an example of it here: https://askubuntu.com/questions/644031/tftpd-hpa-how-can-i-set-blksize-option but I don’t know if ubuntu uses xinetd or something else.

@pjb1983 What version number of ubuntu? 20.04?

[PJB1983]

@george1421

Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04

[george1421]

@pjb1983 So following the link I provided did you try to reduce the maximum packet size to something below 1400 (the mtu)? Something like 1385 would be a start.

[PJB1983]

@george1421 i’ve changed the block size and rebooted the device.
But that did not solve the problem.
If i ask meraki support to increase the MTU for the site-to-site VPN, will that solve it?

[PJB1983]

@george1421 the answer from Meraki Support:

The two sites are using MTU 1432 size due to protocol overhead. This is the recommended value. The best option here is to adjust the MTU of the PXE. Increasing the MTU would result in an MTU greater than 1500 which may lead to fragmentation. Also, Meraki does not support Jumbo size MTU. Should you require further assistance, please do not hesitate to reach out to us asap.

[george1421]

@pjb1983 I could have almost predicted that. Changing the VPN MTU is not really an option.

So you updated the max size from the tftp server. Lets see what the tftp server is telling the target computer.

Lets use this tutorial but for the port capture just use port 69

https://forums.fogproject.org/topic/9673/when-dhcp-pxe-booting-process-goes-bad-and-you-have-no-clue

With the reduced max block size the tftp server should report the size you set in the configuration. You can review the pcap with wireshark.

[Sebastian Roth]

@PJB1983 Just so we get the full picture… So you have another router or layer-3 switch between the PXE booting host and the VPN gateway?

In the best case the two communication partners should adjust MTU/packet size according to response from intermediate gateways and we might find out why this isn’t working properly by looking at the packet dump.

When capturing the traffic please use the filter port 69 or icmp so we get that part as well. Would be great if you could upload the PCAP and post a link here or via private message to George and me.

[PJB1983]

@george1421 then i’ll get

18:47:45.022319 IP 192.168.5.121.2070 > 95.0.0.85.69: 30 RRQ “undionly.kpxe” octet tsize 0
18:47:45.061293 IP 192.168.5.121.2071 > 95.0.0.85.69: 35 RRQ “undionly.kpxe” octet blksize 1456

[PJB1983]

@sebastian-roth can you see something in the file attached?

N_EKS - appliance_MX-MX64 N_EKS_IF-LAN.pcap

[george1421]

@pjb1983 said in Is it possible to connect to the Fog server remotely?:

octet blksize 1456

This is the problem. We need a value less than 1400. I have a ubuntu 20.04 fog server in my home lab. I’ll work with it tonight to see if I can get that block size below 1400. I know we’ve seen this issue before, because the first thing that came to my mind was network MTU.

[PJB1983]

@george1421 THX!

[Sebastian Roth]

@pjb1983 said in Is it possible to connect to the Fog server remotely?:

octet blksize 1456

This is a TFTP thing. Nothing to do with MTU really.

In that PCAP we only see the request packts. Where did you capture those? Can you capture right on your FOG server so we see the reuqests coming in and answers going out? As well we might see ICMP packets from the gateway in case.

If you capture on the gateway you might need to use a different filter as TFTP uses random high ports for the actual data channel. You might use the filter host x.x.x.x and udp or icmp and put in the FOG server IP.

[george1421]

@sebastian-roth said in Is it possible to connect to the Fog server remotely?:

octet blksize 1456

This is a TFTP thing. Nothing to do with MTU really.

I may be off on this, but the tftp client asks the FOG server for the file and what the block size is. Since the block size is larger than the MTU the block will need to be fragmented between two packets. The fragmented udp packets are what the remote client is choking on. At least this is what I’m thinking.

In the pcaps that have been done so far, we see the target computer getting the right pxe boot file name and it asking the fog server for that file and the fog server sending that file but we are seeing a tftp timeout (file never arrived)

I’m not sure where the OP got this pcap from but the numbers align

13:03:04.667062 IP 192.168.5.121.2071 > 95.0.0.85.69: 35 RRQ “undionly.kpxe” octet blksize 1456
13:03:04.692814 IP 95.0.0.85.52966 > 192.168.5.121.2071: UDP, length 15
13:03:04.693826 IP 192.168.5.121.2071 > 95.0.0.85.52966: UDP, length 4
13:03:04.718623 IP 95.0.0.85.52966 > 192.168.5.121.2071: UDP, bad length 1460 > 1400
13:03:04.718623 IP 95.0.0.85 > 192.168.5.121: ip-proto-17
13:03:05.719924 IP 95.0.0.85.52966 > 192.168.5.121.2071: UDP, bad length 1460 > 1400

We know the tftp bock size is sending 1456 and we are getting a bad length 1460 > 1400. The numbers seem to align. Then again, I’ve been wrong too many times to count before, so…

[george1421]

@pjb1983 Well I have success, but still confused why the instructions were wrong. For background the tftp service can be managed by xinetd (normal) but it can also be launch by systemctl directly. It appears with ubuntu 20.04 both methods are setup but only the service route is configured to run.

So exit /etc/default/tftpd-hpa

Make the tftp options look like this:

TFTP_OPTIONS="-s --blocksize 1368"

Once that is done restart the tftp service
sudo systemctl restart tftpd-hpa

Looking at the pcap I see it working as intended (may not be fixing your problem but the block site being adjusted according to the settings).

I can see the client asking for a 1468 block size. The tftp server saying ok but the block site is 1368. Then it sends only 1368 bytes at a time.

So why 1368. That is the mtu 1400 - 32 bytes for the ethernet header == 1368.