Multi-Network, Multi-NIC fog server config help

[lenels2eng]

Hello all and thank you in advance for any assistance that may be provided Past experience with this group has been outstanding!!!

So here’s my setup: I have a FOG server VM running on my ESXi server. I have multiple networks connected to the ESXi host that serve multiple labs. Mgmt is on NIC 0, Lab1 is on NIC 1, Lab2 is on NIC2, etc…I have connected vNICs to my FOG VM for each lab network and can ping the FOG server from each network. I can also ping into the lab networks from the FOG VM so I’m fairly certain the networking is correct.

I can TFTP boot from the default/main/Mgmt network but not from any of my lab networks. I get the FOG white screen with menu options.

When I try from the lab network there is no TFTP response. After doing some research in the forums it seems like I need to add a storage node with the labnetwork IP but the same image location??? Just looking for confirmation that is indeed the trick to using one FOG server on many networks.

Thanks!!!

[lenels2eng]

Tried adding a storage node…No dice…

[Tom Elliott]

@lenels2eng forgive me if I’m not understanding correctly.

What I am gathering is you have a single fog server. That server has multiple nics. Your networking appears to work correctly in that your fog server can ping each of the different networks. Your different networks can also ping the fog server. Here’s my first question: Is the individual networks pinging the fog server at the same ip, or are they pinging the fog server to the associated nic for that network?

My second, what is providing dhcp for each of these networks? If the fog server is doling out the addresses then for each network fog is managing you’ll need a bootserver and filename entry in the related dhcp configuration. If another item is handling dhcp, then that will need to tell those networks where to get the boot information.

My third, if the idea is a single bootserver across vlans, are your switches configured for IP address helping?

[Tom Elliott]

@lenels2eng storage nodes will help in separating the images for the different networks, but will not be helpful in regards to network/tftp booting.

[Sebastian Roth]

@lenels2eng said in Multi-Network, Multi-NIC fog server config help:

I can TFTP boot from the default/main/Mgmt network but not from any of my lab networks.

FOG is not made to serve different networks on one server. While some parts of FOG can handle several networks fairly well (e.g. the apache webserver for the web UI) other services need special configuration and some simply cannot be made to work as far as I know. For this reason FOG does not officially support multihomed network setup.

I am not saying it’s absolutely impossible to achieve but it’s not standard and we don’t have a step by step guide on how to make it work. Definitely needs some manual configuration and fiddling and possibly even changes in the code. If you are keen to get this to work then feel free to play with it and see how far you get. We are happy to answer specific questions when you get stuck at some point. The starting point would be PXE boot. Is FOG your DHCP server or do you have another server handing out IPs and PXE boot information to your lab networks? If FOG is your DHCP then consult /etc/dhcp/dhcpd.conf (or /etc/dhcpd.conf depending on the Linux OS you have FOG installed on) and add more of those subnet definitions to match your needs:

subnet 192.168.1.0 netmask 255.255.255.0 {
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 192.168.1.10 192.168.1.254;
    next-server 192.168.1.1;
...

Note: In default configuration FOG has the next-server parameter outside the subnet definition. AFAIK you need to move that into the subnets as each has a different IP in your case.

When I try from the lab network there is no TFTP response.

Ah, just read this when I had written the stuff above. Never mind, I guess you have a different DHCP server for your labs already setup. Then you need to check that service. Run ss -antul | grep 69 to see if TFTP is running on your FOG server and if it’s set to listen on one IP only. If you see ... *:69 ... that would mean it should listen on all IPs. Then start testing from your lab networks: https://wiki.fogproject.org/wiki/index.php/Unable_to_connect_to_TFTP

What Tom noted is that you usually can setup FOG to serve multiple networks if those networks are not all connected to the FOG server directly but behind a router. So the FOG server only has one IP and all clients connect to that IP through a router. For that to work you usually need to setup DHCP relaying and such things.

[lenels2eng]

Thanks for the replies! DHCP is handled by another device and the correct 66 and 67 entries are there. Yes, the FOG server has multiple NICs, one on each network. Each NIC has an IP on that network. lab1 172.16.111.5, lab2 172.16.112.5, etc.

@Tom-Elliott - The idea is to have one FOG server service 3 networks, each are both physically and logically separate. Each network has it’s own cable plant that terminate at a switch for that network which then connects to a router. The only point of confluence is at the router. The routing is configured that none of the labs can talk to any of the other labs, only out to the firewall and internet.

@Sebastian-Roth - yes, that command returned *:69 so I think it’s listening on all IPs. I only have a few hours to work on it this morning until I have to turn WDS back on to get ready for next week. I’ll go through the testing TFTP from the lab networks you linked to and see if I can make any progress.

I read a while ago, and of course can’t find it when I need it, there are a bunch of people using FOG in a similar setup in the EDU space. I think they were doing multiple labs off one server but passing the TFTP boot only to one IP with DHCP relay as you mentioned as well.

I’m hoping to get this going as I have two scenario’s where this would be AMAZING to get implemented.

Again, thanks guys for being quick and responsive on these forums. MAJOR bonus for using FOG is the relevant help from the guys in the middle of it!!

[Sebastian Roth]

@lenels2eng said:

Each network has it’s own cable plant that terminate at a switch for that network which then connects to a router.

From this it sounds as if you wouldn’t need to have all the lab networks directly connected to your FOG server. But I am still not sure if I get this right. Where exactly is your FOG server (or well, the ESXi server) connected? Does it have X (number of labs) physical connections to those lab switches?

[lenels2eng]

@Sebastian-Roth yes, the esxi host has a NIC on each network. I am attempting to get the network to allow access to an single IP in the management for each lab so I can just use the 066 DHCP option to point to the interface that works for TFTP which I believe is the way the guys that are doing similar setups are making it work.