RancherOS and Fog

george1421

@uberthoth I would set it up akin to this: https://forums.fogproject.org/post/140524

But what you have setup should work too. Just make sure that releases.rancher.com is reachable from the pxe booting client. iPXE should be able to receive that file from a foreign address as long as it can do a dns lookup.

BUT as I said earlier copy the files down to your local FOG server so that stuff is being transferred locally and not across the internet.

uberthoth

@sebastian-roth as far as dhcp/dns/gateway goes, it is the fog server itself that is dhcp server, and like I said networking works fine on the fog server and all other VMs that are spawned on that network (and also got dhcp leases from the fog server).

Where do I check on this configuration? /etc/dnsmasq.d only has a lxd file in it, with these contents:

/etc/dnsmasq.d# cat lxd 
# Tell any system-wide dnsmasq instance to make sure to bind to interfaces
# instead of listening on 0.0.0.0
# WARNING: changes to this file will get lost if lxd is removed.
bind-interfaces
except-interface=lxdbr0

george1421

@uberthoth What are you using dnsmasq for? Its not configured to do anything. If this is a typical FOG server setup and you are using the fog install dhcp server then isc-dhcp server should be used. DNSMASQ in the case of isc-dhcp, dnsmasq should not be used or needed.

Are you using the FOG server as a router between what I’m guessing is an imaging network and the business network?

You need to go into a bit more detail on how you have your network setup so we can give you ideas to test.

uberthoth

@george1421 My question was where do I configure DHCP for fog. My assertion was that dnsmasq was not being used as it only had the lxd file in there, I am not using or configuring dnsmasq at all, and forgive my assumption that fog was using it (it seemed to me to be a likely choice for such a task, I was wrong). Do I use the GUI in fog, or should I be editing files in /etc/dhcp directly? There is no business network, this is merely a sandbox setup for testing.

My test environment consists of a virtual LAN created on a KVM linux host, this network was created by right clicking on the KVM host in virt-manager and then clicking ‘details’ and then ‘virtual networks’ and clicking the plus sign (Add Network) there. The resulting xml for this network is:

<network connections="2">
  <name>fog</name>
  <uuid>6cac86e9-ff5c-4b27-bc27-41f517524341</uuid>
  <forward mode="nat">
    <nat>
      <port start="1024" end="65535"/>
    </nat>
  </forward>
  <bridge name="virbr1" stp="on" delay="0"/>
  <mac address="52:54:00:95:b5:4f"/>
  <domain name="fog"/>
  <ip address="192.168.89.1" netmask="255.255.255.0">
  </ip>
</network>

/etc/dhcp/dhcpd.conf <-- I have not edited this file, but am including it for reference

# DHCP Server Configuration file\n#see /usr/share/doc/dhcp*/dhcpd.conf.sample
# This file was created by FOG
#Definition of PXE-specific options
# Code 1: Multicast IP Address of bootfile
# Code 2: UDP Port that client should monitor for MTFTP Responses
# Code 3: UDP Port that MTFTP servers are using to listen for MTFTP requests
# Code 4: Number of seconds a client must listen for activity before trying
#         to start a new MTFTP transfer
# Code 5: Number of seconds a client must listen before trying to restart
#         a MTFTP transfer
option space PXE;
option PXE.mtftp-ip code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sport code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;
option arch code 93 = unsigned integer 16;
use-host-decl-names on;
ddns-update-style interim;
ignore client-updates;
# Specify subnet of ether device you do NOT want service.
# For systems with two or more ethernet devices.
# subnet 136.165.0.0 netmask 255.255.0.0 {}
subnet 192.168.89.0 netmask 255.255.255.0{
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 192.168.89.10 192.168.89.254;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 192.168.122.1;
    option domain-name-servers 127.0.0.53;
    next-server 192.168.89.2;
    class "Legacy" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
        filename "undionly.kkpxe";
    }
    class "UEFI-32-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-32-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-64-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00007";
        filename "ipxe.efi";
    }
    class "UEFI-64-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00008";
        filename "ipxe.efi";
    }
    class "UEFI-64-3" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00009";
        filename "ipxe.efi";
    }
    class "SURFACE-PRO-4" {
        match if substring(option vendor-class-identifier, 0, 32) = "PXEClient:Arch:00007:UNDI:003016";
        filename "ipxe.efi";
    }
    class "Apple-Intel-Netboot" {
        match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                filename "ipxe.efi";
            }
        }
    }
}

All other VMs on the fog network have full functioning internet access on the fog network, and received their leases from the fog dhcp server (and most only have one NIC adapter on that network). The fog VM itself is receiving a DHCP lease from my router through a bridge (br0) on the KVM host, and all other VMs on that same bridge (peer to the fog vm, and not inside the fog network) all have full functioning internet and so does the fog VM itself.

I was able to download the vmlinuz and initrd and place them in the root for fog. However, the wiki pages contain some misleading information. $boot_url does not work as implied on this page. I had to use the $fog-ip etc variables from this deprecated page. Which makes me wonder am I running an out of date version? It says 1.5.9.

george1421

@uberthoth OK let me start off with, I have not setup a KVM hypervisor so I can’t be helpful here.

Looking at the ics-dhcp configuration that should boot your target computer into the FOG iPXE menu. If you get to that point we are golden.

I do see one flaw in the design but lets work through it.

Now I’m trying to understand the networking part. The isc-dhcp server configuration will give the PXE booting clinets an IP address in the 192.168.89.10 to 192.168.89.254 range. Your subnet is 192.168.89.0/24. How do the clients get off (leave) the 192.168.89.0/24 subnet? Your defined router is 192.168.122.1 in the config file is, but the clients are on 192.168.89.0/24 and have no way to talk to 192.168.122.1 (because its on a different subnet). You need to have a router/gateway on 192.168.89.0/24 to be able to leave that subnet. Just be aware its possible to make the FOG server this gateway/router with some adjustments.

The second issue you have is your DNS entry is pointing back at itself (sounds like a ubuntu thing). You need to ensure that the pxe booting computer is getting a valid DNS server address that can resolve your external domain name.

Sebastian Roth

@uberthoth said in RancherOS and Fog:

subnet 192.168.89.0 netmask 255.255.255.0{
option subnet-mask 255.255.255.0;
range dynamic-bootp 192.168.89.10 192.168.89.254;
default-lease-time 21600;
max-lease-time 43200;
option routers 192.168.122.1;
option domain-name-servers 127.0.0.53;

Well, I see two issues with that isc-dhcp configuration.

option routers 192.168.122.1; is not within the same subnet “192.168.89.0”. How would clients be able to reach that router at all?
option domain-name-servers 127.0.0.53 is pointing to a localhost DNS server (used by Ubuntu).

You will need to correct those two things and restart isc-dhcp service (or your whole FOG server) for clients to be able to communicate with external hosts at all.

When you change those things in /etc/dhcp/dhcpd.conf you also want to edit /opt/fog/.fogsettings and get that right. Because next time you run the FOG installer for an update it would update dhcpd.conf with the wrong values again.

uberthoth

@sebastian-roth I’m not certain how the 192.168.122 network got mixed in there, but once I changed the option routers to 192.168.89.1 and it still does not work.

I’ve been through the installer a half dozen times now in the past couple of days, and I might just be misunderstanding some of the install questions. When it asks me about the default interface, my assumption is that it is asking which to use as WAN, is this correct? If it is WAN, then the ‘router on the DHCP server’, would be the upstream router. However, a few of the installs I have made the fog LAN interface the default interface in which case it does default again to the upstream router (this is where 192.168.122.1 came in my first install), I expect that in this case the upstream router should be 192.168.89.1.

So to try things a bit differently I made a new fog server with two interfaces one on br0 where it receives a 10.1.2.0/24 address from my ISPs modem/router, and another interface on br2, an isolated network interface, on the host, that fog can have to itself, for an image network. I assign it a static IP 10.88.88.1. Then I run through the fog install and provision a VM onto this same br2 interface and no other interface. The VM does indeed get a lease, e.g. 10.88.88.12, and can communicate to other addresses on that same net, but cannot route to any known ip addresses at large (8.8.8.8 etc).

That section of dhcpd.conf looks like this now:

subnet 10.88.88.0 netmask 255.255.255.0{
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 10.88.88.10 10.88.88.254;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 10.88.88.1;
    option domain-name-servers 127.0.0.53;
    next-server 10.88.88.1;

I’ve tried adding a section to ignore the br0 network

subnet 10.1.2.0 netmask 255.255.255.0 {}

netplan on the fog server:

network:
        version: 2
        renderer: networkd
        ethernets:
                eth0:
                        dhcp4: yes
                eth1:
                        addresses:
                                - 10.88.88.1/24

In none of these cases am I able to get fog to route network upstream to it’s dhcp clients in either of the fog controlled networs br2(10.88.88.0/24), and fog(192.162.89.0/24). I suspect that any of my previous assertions as to machines getting functioning internet were dual interface machines (i.e. they had a NIC in both of the fog(192.162.89.0/24), and the default (192.162.122.0/24) networks). What am I doing wrong in during the install process? I have a feeling it comes down to the default interface and the router option, but I’m open to suggestions.

george1421

@uberthoth OK so this appears to now be down to a routing problem.

What I want you to do is take a running windows computer and plug it into the imaging network.

I want you to make sure the dhcp settings defines the default gateway as 10.88.88.1 also make sure the dns server is 10.88.88.1.

When those are checked then make sure you have turned your FOG server into a router by keying in this command on the fog server console. You need to be root or sudo to run this command echo 1 > /proc/sys/net/ipv4/ip_forward That tells the linux kernel to forward packets between its interface. Since we are making this change temporary after a reboot the fog server will revert to a workstation again. First we want to test second we make it forever.

Now from the fog server ensure you can ping the internet. If that is OK, then the last thing you need to do is on the upstream router (default router for FOG) you need to create a static IP route to the 10.88.88.0/24 subnet via the business LAN interface of the FOG server. This way the pxe booting computers know how to get to the fog server and the fog server to the upstream router. Then when data comes back from the internet the upstream router knows to send data to 10.88.88.0/24 via the FOG server. Now from the windows computer on the imaging subnet, ping 8.8.8.8 and 1.1.1.1 You should get a response. If that works then you have basic routing in place.

The last part you will need to work on is DNS, but first test basic routing.

Sebastian Roth

@uberthoth said in RancherOS and Fog:

I’ve been through the installer a half dozen times now in the past couple of days, and I might just be misunderstanding some of the install questions. When it asks me about the default interface, my assumption is that it is asking which to use as WAN, is this correct?

I know this bit can be confusing. While FOG needs to have an internet connection (and therefore an upstream gateway) to be able to install packages from the official distro repos and download files from the FOG website it doesn’t care to know about that gateway, it just relies on it working correctly.

So the question on network interface is meant as “Which interface and IP should FOG use to do the imaging?” - so in a setup with more than one interface, usually not the upstream interface.

Now the question about nameserver and gateway is only for the DHCP service configuration. It’s not for the FOG server itself. The installer does not change those Linux network settings for you. You need to get this all right before running the installer.

@uberthoth said in RancherOS and Fog:

I’m not certain how the 192.168.122 network got mixed in there, but once I changed the option routers to 192.168.89.1 and it still does not work.

It’s either DNS/nameserver information in the dhcpd.conf still wrong or IP forwarding not activated (see George’s post) or firewall rules blocking the forwarding.

uberthoth

@sebastian-roth Thanks for clearing up the ‘default interface’ question. I was maybe rabbit holing that a bit.

For certain ip_forward is set on all fog hosts that I have spun up in the past week:

cat /proc/sys/net/ipv4/ip_forward
1

As a sanity check, I installed fog on a fedora host as well. And low and behold the fedora host worked the very first time! That solves my needs and I can move forward. However, I do still have the ubuntu VMs dormant that I can spin up and do testing on, and I would like to contribute any knowledge gained here to the wiki.

Here is the script I use to setup masquerading on iptables:

#!/bin/bash
WAN_INTERFACE=eth2

check_ip_forwarding () {
  if [[ $(cat /proc/sys/net/ipv4/ip_forward) ]]; then
    echo  'ip forwarding already enabled'
  else
    #echo 1 > /proc/sys/net/ipv4/ip_forward
    sysctl -w net.ipv4.ip_forward=1
    echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.d/net.ipv4.ip_forward.conf
    echo  'ip forwarding enabled'
  fi
}

add_network_to_forwards () {
  LOCAL_NETWORK_CIDR=$1
  echo "Adding $LOCAL_NETWORK_CIDR to network forwards."
  sudo iptables -I FORWARD -o $WAN_INTERFACE -s $LOCAL_NETWORK_CIDR -j ACCEPT
  sudo iptables -I INPUT -s $LOCAL_NETWORK_CIDR -j ACCEPT
}

check_ip_forwarding
sudo iptables -t nat -A POSTROUTING -o $WAN_INTERFACE -j MASQUERADE

add_network_to_forwards 10.88.88.0/24
add_network_to_forwards 10.88.89.0/24
add_network_to_forwards 192.168.89.0/24

and here is my current netplan from the ubuntu host:

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
    eth1:
      addresses:
        - 192.168.89.2/24
    eth2:
      dhcp4: true
    eth3:
      addresses:
        - 10.88.88.1/24
    eth4:
      addresses:
        - 10.88.89.1/24

where:
eth0 = default virtual network on KVM
eth1 = fog virtual network on KVM
eth2 = ISP/router LAN
eth3 = br2 on KVM host hardware interface
eth4 = br3 on KVM host hardware interface

/etc/dhcp/dhcpd.conf:

option space PXE;
option PXE.mtftp-ip code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sport code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;
option arch code 93 = unsigned integer 16;
use-host-decl-names on;
ddns-update-style interim;
ignore client-updates;
subnet 10.1.2.0 netmask 255.255.255.0 {}
subnet 192.168.89.0 netmask 255.255.255.0 {
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 192.168.89.10 192.168.89.254;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 192.168.122.1;
    option domain-name-servers 127.0.0.53;
    next-server 192.168.89.2;
    class "Legacy" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
        filename "undionly.kkpxe";
    }
    class "UEFI-32-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-32-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-64-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00007";
        filename "ipxe.efi";
    }
    class "UEFI-64-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00008";
        filename "ipxe.efi";
    }
    class "UEFI-64-3" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00009";
        filename "ipxe.efi";
    }
    class "SURFACE-PRO-4" {
        match if substring(option vendor-class-identifier, 0, 32) = "PXEClient:Arch:00007:UNDI:003016";
        filename "ipxe.efi";
    }
    class "Apple-Intel-Netboot" {
        match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                filename "ipxe.efi";
            }
        }
    }
}

subnet 10.88.88.0 netmask 255.255.255.0 {
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 10.88.88.3 10.88.88.254;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 10.88.88.1;
    option domain-name-servers 8.8.8.8;
    next-server 10.88.88.1;
    class "Legacy" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
        filename "undionly.kkpxe";
    }
    class "UEFI-32-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-32-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-64-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00007";
        filename "ipxe.efi";
    }
    class "UEFI-64-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00008";
        filename "ipxe.efi";
    }
    class "UEFI-64-3" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00009";
        filename "ipxe.efi";
    }
    class "SURFACE-PRO-4" {
        match if substring(option vendor-class-identifier, 0, 32) = "PXEClient:Arch:00007:UNDI:003016";
        filename "ipxe.efi";
    }
    class "Apple-Intel-Netboot" {
        match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                filename "ipxe.efi";
            }
        }
    }
}

subnet 10.88.89.0 netmask 255.255.255.0 {
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 10.88.89.3 10.88.89.254;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 10.88.89.1;
    option domain-name-servers 8.8.8.8;
    next-server 10.88.89.1;
    class "Legacy" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
        filename "undionly.kkpxe";
    }
    class "UEFI-32-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-32-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-64-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00007";
        filename "ipxe.efi";
    }
    class "UEFI-64-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00008";
        filename "ipxe.efi";
    }
    class "UEFI-64-3" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00009";
        filename "ipxe.efi";
    }
    class "SURFACE-PRO-4" {
        match if substring(option vendor-class-identifier, 0, 32) = "PXEClient:Arch:00007:UNDI:003016";
        filename "ipxe.efi";
    }
    class "Apple-Intel-Netboot" {
        match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                filename "ipxe.efi";
            }
        }
    }
}

VMs joining the network do get DHCP leases, and they can communicate with other machines on the network they join, but the ubuntu VM does not seem to be routing upstream despite me setting the above rules in iptables and ensuring ip_forward is on, and I’ve even chucked in a iptables -F before everything to bang on things with a bigger hammer.

EDIT: the original ubuntu VM was perhaps laden with too much experimentation, after creating a new VM and applying the above finalized configs everything is working now.

Cheers, I have a fully functioning fog to assimilate all the things and I will contribute to the wiki once I repeat this a few times to ensure I’ve got everything working properly.

uberthoth

If you want to make god laugh just type finalized in a forum post.

Back to rancher, everything works fine with a single network defined in dhcpd.conf. ipxe has network upstream, gets the kernel, initrd, and the defined cloud-init file.

However, if I do run with all three networks defined as above, the rancher VM gets the ‘next’ and ‘router’ options from the 192.168.89.0 network, despite having gotten an address defined in the 10.88.88.0 network. Attaching

Notice how it received the ip 10.88.88.15 but got the gw and ‘next server’ from the 192.168.189.0 network.

Here is my currenct dhcpd.conf off of that fog VM:

# DHCP Server Configuration file\n#see /usr/share/doc/dhcp*/dhcpd.conf.sample
# This file was created by FOG
#Definition of PXE-specific options
# Code 1: Multicast IP Address of bootfile
# Code 2: UDP Port that client should monitor for MTFTP Responses
# Code 3: UDP Port that MTFTP servers are using to listen for MTFTP requests
# Code 4: Number of seconds a client must listen for activity before trying
#         to start a new MTFTP transfer
# Code 5: Number of seconds a client must listen before trying to restart
#         a MTFTP transfer
option space PXE;
option PXE.mtftp-ip code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sport code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;
option arch code 93 = unsigned integer 16;
use-host-decl-names on;
ddns-update-style interim;
ignore client-updates;
# Specify subnet of ether device you do NOT want service.
# For systems with two or more ethernet devices.
# subnet 136.165.0.0 netmask 255.255.0.0 {}
subnet 10.0.23.0 netmask 255.255.255.0 {}
subnet 192.168.122.0 netmask 255.255.255.0 {}
subnet 192.168.89.0 netmask 255.255.255.0 {
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 192.168.89.10 192.168.89.254;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 192.168.89.2;
    option domain-name-servers 127.0.0.53;
    next-server 192.168.89.2;
    class "Legacy" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
        filename "undionly.kkpxe";
    }
    class "UEFI-32-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-32-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-64-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00007";
        filename "ipxe.efi";
    }
    class "UEFI-64-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00008";
        filename "ipxe.efi";
    }
    class "UEFI-64-3" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00009";
        filename "ipxe.efi";
    }
    class "SURFACE-PRO-4" {
        match if substring(option vendor-class-identifier, 0, 32) = "PXEClient:Arch:00007:UNDI:003016";
        filename "ipxe.efi";
    }
    class "Apple-Intel-Netboot" {
        match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                filename "ipxe.efi";
            }
        }
    }
}

subnet 10.88.88.0 netmask 255.255.255.0 {
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 10.88.88.3 10.88.88.254;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 10.88.88.1;
    option domain-name-servers 8.8.8.8;
    next-server 10.88.88.1;
    class "Legacy" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
        filename "undionly.kkpxe";
    }
    class "UEFI-32-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-32-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-64-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00007";
        filename "ipxe.efi";
    }
    class "UEFI-64-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00008";
        filename "ipxe.efi";
    }
    class "UEFI-64-3" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00009";
        filename "ipxe.efi";
    }
    class "SURFACE-PRO-4" {
        match if substring(option vendor-class-identifier, 0, 32) = "PXEClient:Arch:00007:UNDI:003016";
        filename "ipxe.efi";
    }
    class "Apple-Intel-Netboot" {
        match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                filename "ipxe.efi";
            }
        }
    }
}

subnet 10.88.89.0 netmask 255.255.255.0 {
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 10.88.89.3 10.88.89.254;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 10.88.89.1;
    option domain-name-servers 8.8.8.8;
    next-server 10.88.89.1;
    class "Legacy" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
        filename "undionly.kkpxe";
    }
    class "UEFI-32-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-32-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-64-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00007";
        filename "ipxe.efi";
    }
    class "UEFI-64-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00008";
        filename "ipxe.efi";
    }
    class "UEFI-64-3" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00009";
        filename "ipxe.efi";
    }
    class "SURFACE-PRO-4" {
        match if substring(option vendor-class-identifier, 0, 32) = "PXEClient:Arch:00007:UNDI:003016";
        filename "ipxe.efi";
    }
    class "Apple-Intel-Netboot" {
        match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                filename "ipxe.efi";
            }
        }
    }
}

Sebastian Roth

@uberthoth A quick search on the web revealed this topic: https://bugs.launchpad.net/maas/+bug/1521618 - post #17 is the most interesting one!

With multiple subnets, PXE DHCP clients would receive the router of the first
subnet in the configuration file regardless of which subnet they actually
received a lease from.
…
https://lists.isc.org/pipermail/dhcp-users/2011-September/014001.html

As far as I understand it the class definition within the subnet is somehow also overwriting the router/gateway information for a client. So when dhcpd matches against the vendor-class definitions it will always match in the first subnet and therefore also return that gateway information. Kinda strange but that’s how I understand this.

To fix this you’d move the class definitions out to the global scope. Just have one batch of those globally instead of one in each subnet definition.

In FOG we usually never have more than one subnet and really, FOG doesn’t handle multiple subnets well (e.g. multicasting and other things!). So yes, the dhcpd.conf we generate and have in the wiki might not be an ideal start if you want to extend it to provide for more than subnet.

uberthoth

@sebastian-roth Thanks for the extra info, especially considering the multicasting, I think it might be safe to call the multiple subnets out of scope. Just give fog it’s own network, and I can have another machine be in charge of the other nets. It might be preferable to being doing what I’m doing anyhow with a bond or a bridge anyhow and leave it all as one subnet. So I think that does wrap this one up. I’ll give the two rancher parameter entries:

#!ipxe
set base-url http://releases.rancher.com/os/latest
kernel ${base-url}/vmlinuz rancher.state.dev=LABEL=RANCHER_STATE rancher.state.autoformat=[/dev/sda] rancher.cloud_init.datasources=[url:https://raw.githubusercontent.com/EXAMPLEgithubuser/EXAMPLErepo/master/user_data]
initrd ${base-url}/initrd
boot

Also, an entry for the local entries where everything must be downloaded from the fog server itself, in my case I made a directory called tertiary and placed a rancher directory in there with all the downloaded files.

#!ipxe
set base-url http://${fog-ip}/${fog-webroot}/tertiary/rancher
kernel ${base-url}/vmlinuz rancher.state.dev=LABEL=RANCHER_STATE rancher.state.autoformat=[/dev/sda] rancher.state.wait rancher.cloud_init.datasources=[url:${base-url}/user_data]
initrd ${base-url}/initrd
boot

george1421

@uberthoth Now in regards to your iPXE menu configuration. If you want to see the context of what your menu item will run in open a browser to this url, http://<fog_server_ip/fog/service/ipxe/boot.php?mac=00:00:00:00:00 This is the text that creates the iPXE menu. Specifically look at the top of the document and see how the FOG server’s IP address is assigned to which variables. HINT: You may want to move the rancher boot files out of fog file path into a different one like /os instead of /fog.

RancherOS and Fog

261

11.7k

17.1k

154.7k