I got it working! Eureka!
Thank you for the tips. I found the iPXE documentation and the ${net0/mac} came in handy!
I have used buildroot before, on those BeagleBones / Raspberry Pie systems. It does have a learning curve and I am certain you can use it for what I want to do.
But…I figured out how to do exactly what I wanted to do!!! I will share because I can tell you guys are oolging to give this a shot yourselves
Step 0: Get PXE to boot a simple NFS volume first
I strongly recommend you follow George’s very helpful guides (Thanks George!!!): https://forums.fogproject.org/topic/10944/using-fog-to-pxe-boot-into-your-favorite-installer-images/17
I followed the instructions for booting the Ubuntu Live CD, which coincidentally you will need twice if you follow in my footsteps. If you cannot get this working, stop and get this working! There is no point in making things more complicated if you cannot boot prepackaged ISOs.
Step 1: Create a template machine
So first things first, you need an installed copy of Ubuntu or whatever OS/distro you are using. We will template this machine. I spun up a Virtual Machine for this purpose and PXE booted into the Ubuntu Live CD. When installing, use a single partition. The size is irrelevant, so just make it big enough to finish installation. Boot into your installed OS, and get it all set up with whatever you want to snapshot. It is easy to add stuff later, so I wouldn’t lament over this.
Step 2: Set up your NFS share
We need a NFS share to store the image of this system on. It needs to be a read/write NFS share set up somewhere on your network accessible from your client machines. The FOG box is okay; not sure I would use it long term. I set up my nfsroots in /images/dev/nfsroots. I have not yet tinkered with permissions, but you will want to do this at some point because otherwise people can screw with your images over network (as root no less)! The following line works in my /etc/exports file. It is untouched from fog installation.
/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)
Finally, scribble down the mac address of the system you will be imaging from. Make a directory in nfsroots like so: /images/dev/nfsroots/8a:4b:84:c8:e2:b9. This is where we will store the image of this specific system.
Step 3: Clone your template machine
Boot your image system into a live disk. You need something that allows you to drop into terminal, install some basic packages, and run some basic commands. The Ubuntu live CD is perfect. Run these commands:
sudo apt-get install nfs-common # The live CD does not come with the NFS packages preinstalled
sudo mkdir /mnt/sda1
sudo mkdir /mnt/nfs
sudo mount /dev/sda1 /mnt/sda1 # If you booted into the live disk, your hard disk / boot partition probably isn't mounted. So mount it.
sudo mount 192.168.0.10:/image/dev/nfsroots /mnt/nfs # Put the IP of your storage server and your nfsroot path here
cp -a /mnt/sda1/* /mnt/nfs/8a:4b:84:c8:e2:b9/ # The -a is super important. Archive mode. We must preserve the ownership and the SUID bits on file permissions
Once you have done this, you can shutdown the system. Remove the (virtual) hard disk. I would keep it around until you finish in case you need to repeat any steps.
Step 4. Copy the kernel and initial ramdisk from your image
We cannot retrieve these over NFS because these are what set up the NFS connection. If you followed George’s instruction for booting the Ubuntu Live CD, you took the vmlinuz (kernel) and initrd (initial ramdisk) from the Live CD and copied them to /tftpboot on your fog box. DO NOT USE THESE for booting the full OS. It seems like these are hardwired to crack open the squashfs (the compressed read-only filesystem) on the live CD. Instead, get the vmlinuz and initrf.img from the root you just copied. Put them in the tftp folder.
sudo cp /images/dev/nfsroots/8a:4b:84:c8:e2:b9/vmlinuz /tftpboot/xubuntu/18.04.2/
sudo cp /images/dev/nfsroots/8a:4b:84:c8:e2:b9/initrd.img /tftpboot/xubuntu/18.04.2/
Step 5. Set up a new boot profile in FOG
Go to FOG Configuration -> New iPXE Menu
Copy this information in:
Menu Item: NFS Boot by MAC (or whatever you want)
Description: NFS Boot from /images/dev/nsfroot/${net0/mac} (or whatever you want. Note that ${net0/mac} is parsed and will actually display as your system’s MAC address.
Parameters:
kernel tftp://${fog-ip}/xubuntu/18.04.2/vmlinuz # Customize to point to wherever the vmlinuz is you copied in step 4
initrd tftp://${fog-ip}/xubuntu/18.04.2/initrd.img # ditto
imgargs vmlinuz root=/dev/nfs vga=normal nfsroot=${fog-ip}:/images/dev/nfsroots/${net0/mac}/ rw locale=en_US.UTF-8 persistent console=tty0 console=ttyS0,115200
boot || goto MENU
Menu Show with: All Hosts
Leave everything else blank.
imgargs is the super secret sauce to making this all work. These are the kernel parameters that are going to get passed into that kernel you just supplied over tftp. Basically, these instructions tell the kernel where root is. Root is a coming from nfs, The path to the nfs share is blah. root is read/write. We want vga turned on. We omitted quiet and splash; we want to see all the juicy boot progress details flying across the screen. I’ve added the console strings for testing purposes. These will dump the boot progress out on a serial port, which is extremely helpful if you run into problems because every problem is going to be a full blown kernel panic. You won’t be able to scroll up on your vga terminal to see what the error was! Since I am using the KVM hypervisor, I can add a virtual serial port and I connect to it via the appropriate command issued at the hypervisor’s terminal. If you encounter any issues, serial debug is a good place to start.
Step 6. Housekeeping
We have some rough edges that need to be cleaned up, otherwise boot is going to be somewhat turbulent.
open /images/dev/nfsroots/8a:4b:84:c8:e2:b9/etc/fstab from your nfs host machine. This file contains the list of mount points to setup at boot. Remove everything. We are not in Kansas anymore Dorthy! /dev/sda doesn’t exist anymore, and I seriously doubt you want to have a network mounted swap. If you boot hangs for about 90 seconds, it is probably because you forgot to clear this file. You will only see that boot has hung to wait for a mount if you have serial terminal set up; it doesn’t show on vga boot progress.
open /images/dev/nfsroots/8a:4b:84:c8:e2:b9/etc/hostname from your nfs host machine. This is the hostname of your client. Change this if your template might ever be booted in the future (you know it will, so change it!). Change this every time you copy this image for another machine.
Step 7. Boot
Power on your machine, making sure it goes to PXE. Interrupt it to select the boot option we just created. Cross your fingers!
TODO There are some rough edges arround this setup. If you have ideas on how to clean this up, please chime in.
- Default boot option. Those experienced with FOG will know that its not possible to register a system with no hard disk. Therefore, it is not possible to nudge your clients towards this particular boot option we’ve set up. You can go around it backwards and configure every registered machine to do something else, and all unregistered machines will try to boot NFS by MAC address. But, that is backwards. There’s got to be an obvious fix to this.
- Security. Its basically non-existent and I don’t even know where to start. Any keys or passwords we try to push to the clients will be visible on the http server FOG uses to push boot instructions. No-go. We have a whole bunch of spoofable information we could use, but I wager things like MAC address and IP filters are only going to delay an attack. Its not until NFSv4 that proper authentication was really added, and everything shown here uses NFSv3. I really, really want to have a private key stored on each client, burned into the TPM or some other hardware security mechanism where it is never ever getting out. Can iPXE even touch these resources? If I cannot figure out how to use a proper public-private key pair, at least I want to derive the passwords on the clients where a lucky network snoop is required to get them. Ideas?
- Package installer freak out. Snap is completely hosed. Do not use it. Apt expects that it needs to update the initramfs after installing every packages; That is okay…I think. Do I need to watch for the initramfs being updated and clone that over to tftp, or am I correct in assuming that initramfs from tftp is not used after nfsroot is established as real root? Apt also expects to update grub to point to the new initramfs. Grub does not exist anymore so that throws a bunch of ugly errors. To be clear; packages still install correctly and you can use them after reboots.
- No Swap. One of the givens was this system has no hard disk. Therefore there is nowhere to store swap. Unless you mount swap over network, which seems like a really bad idea. There is margin for tuning the kernel memory limits, but I think it should be mostly okay???
- De-duplication. My linux-fu is pretty good, but not complete. I wager many of the directories I copied in the clone never change in the lifetime of the OS. Or, any changes could be easily shared among all of the identically imaged systems without problems. Ideas from somebody who knows more about this?