FOG TFTP Slow over VLANs

[xardoniak]

Hi all,
Just wanting to confirm I haven’t configured something incorrectly inside of FOG.
I have FOG installed on an Ubuntu 18 server VM, on my Server VLAN. When booting my Ubuntu 18 ISO to another VM on the same VLAN, same host, it copies at anywhere from 105MB/s to 150MB/s per the FOG dashboard
When booting the same ISO on a different VLAN, the dashboard shows 3MB/s stable, with the occasional jump to 7MB/s - significantly slower!
I’ve checked SMB crossing VLANs and the speed caps out at 100MB/s, though this is 2 different machines (my gaming PC and Synology NAS)

My firewall is a Sophos XG, I have tried
-Disabling Traffic Shaping - no change (only enabled for WAN)
-Rebooting the Firewall - no change

Here is my network diagram

Not to sure what to check next, just wanting to confirm that there isn’t a cap’d speed in FOG somewhere or if I need to enable something for better performance over subnets.

Edit: I am not apposed to adding NICs to the VM as a work around - might end up doing this…

[Sebastian Roth]

@xardoniak FOG is not limiting the speed I reckon. What I can imagine is this being caused not by the different VLANs but by iPXE drivers. Have you tested with the exact same make, model und firmware version of host in the different VLANs yet?

[xardoniak]

Hi mate,
This is my testing method
FOG is on host “Jarvis”, on Server VLAN

1. Create new VM on Jarvis, with server VLAN
2. Boot Ubuntu ISO, 100+MB/s
3. Cancel install, shut down VM
4. Change VM NIC to Trusted VLAN (not touching the driver)
5. Boot Ubuntu ISO, 3MB/s

Thanks for the help!

[Sebastian Roth]

@xardoniak Well, would have been a valuable information that this is happening in a VM environment. What hypervisor do you use???

[george1421]

@xardoniak I would kind of suspect the router between the server vlan and trusted vlan or the vm host’s network interface.

The other thing to consider is that when you have 2 vms on the same vlan in the same esxi box, the data flowing between the two never leave the esxi virtualized environment. The vSwitch on the esxi box just redirects the traffic between the two VMs, so that data never really “hits the wire”. Where when you go between the vlans the data leaves the esxi box, hits your vlan router and then goes back into esxi box.

Another thought is how are you booting the ubuntu iso image? Are you using memdisk or my tutorial on booting your favorite OS with FOG? The communications protocol being used will have an impact on cross vlan speeds. FOG can use tftp, http, nfs to transfer the files to the target computer. How do you have this setup?

[xardoniak]

@george1421 I followed the Favorite OS guide and when booting to the ISO, it states tftp://10.0.70.10/os/ubuntu etc etc
I was thinking the same but the Firewall / Router is a VM on the same ESXi host. I don’t have these speed issues over SMB though.

[george1421]

@xardoniak Well to start breaking this down, do you have a physical computer on the same subnet you can test with?

Really we have 2 timings to consider. The first is the transfer of the linux kernel and initfs via tftp, then you have the kernel startup to end that timing. Once the kernel is up and running it uses nfs to connect to the FOG server to load the squashfs that has the actual OS image. Once the nfs bit happens that starts the second timing. tftp is the slowest of the protocols but that is only used to get vmlinz and initrd over to the target computer, the rest is done by nfs.