I intend to document the stuff I learned so far in regards to pxe booting and FOG’s interaction.

Before I get into some of the details of what I’ve learned chatting with Tom and through chats with others like Sebastian, I want to cover some of the basics about pxe booting a target computer into the FOG environment.

PXE booting (known then as diskless booting) has been around in the *nix world since the mid to late 1980s. During that time it was known as network bootstrapping. In 1999 Intel defined a standard called “Wired for Management” which defined the structure that is now known as Pre eXecution Environment (PXE). If I remember correctly on-board PXE code was added as part of PC 2001 design specification by Intel (PC 201 specs also meant the death of the ISA bus). Prior to the PC 2001 specification the PXE boot code was typically added to the network adapter in a PXE boot ROM.

Booting the FOG client OS (I still don’t have a good name for the operating system that runs on the target computer, which does the actual work of imaging the target. For this document it will be known as FOS) requires several services to work in concert. If one actor drops the note the whole concert is ruined. To get the FOS system running the following actors are required. DHCP, TFTP, IPXE, and HTTP.

PXE ROM
When the target computer is powered on and pxe booting is selected, the first thing the PXE code on the target does is attempt to acquire an IP address from the local DHCP server. Once the target has an IP address, it again queries the DHCP server for its environment. This is where the target computer learns about 2 specific and important dhcp options. These important dhcp settings are option 66 (Boot server IP) [in our case the FOG server] and option 67 (Boot file name) [in our case the iPXE network boot firmware]. These two options are passed into the PXE client’s network boot code. The typical network boot code reaches out to device referenced by dhcp option 66 (Boot server IP) using the TFTP (trivial file transfer protocol) to download the file referenced by dhcp option 67 (Boot file name). Once this file is downloaded from the tftp server the pxe boot strap code then chain executes to the file that was downloaded. In the case of the FOS booting, the file that is downloaded is a flavor of iPXE http://ipxe.org/ which is intended to extend the simple capabilities of the built in PXE environment found on the target computer’s NIC rom.

iPXE
The iPXE network boot firmware then reconnects to the device pointed to by dhcp option 66 (i.e. the FOG server) and downloads its configuration file to know what to do next. The configuration file iPXE retrieves (via tftp) is the file default.ipxe. This config file instructs the iPXE network boot firmware to reconnect to the FOG server this time using the http protocol to access the boot.php web page. The iPXE environment also passes the mac address of the target computer to the boot.php page as a parameter. This web page then interacts with the FOG database to constructs the FOG boot menu. In its simplest form the iPXE network boot firmware is an executing operating system with a limited command set. You’ll notice that the PXE boot rom built into the computer will query for a IP address from your dhcp server, and then when the iPXE network boot firmware starts, it will again requrey for an IP address form your dhcp server. And when FOS starts it will again require for an IP address from your dhcp server. It sounds redundant, but each environment is an independent operating system.

FOG Boot Menu
The fog boot menu is created based on the current state of the target computer. If there is a pending job, the target computer is instructed to execute the job. If no job is pending the default FOG menu is displayed. The fog boot menu is constructed of iPXE commands to execute once the IT technician selects an option. For example if you were to look at the iPXE commands for Quick Registration. You would see the iPXE instruction to download the bzImage(32) kernel (the FOS operating system) and execute it with certain parameters as well as the instruction to download the inits(_32).xz (the FOS virtual hard drive). Both of these files will be downloaded to the target computer’s RAM over the http protocol and then executed.

FOS
Once both files required to boot FOS has been transfered to the target computer, the iPXE code chain boots the bzImage (FOS kernel) which begins execution and starts the FOS booting. During the booting process the FOS kernel connects to the downloaded virtual hard drive and starts the init process to boot the entire FOS environment. Again FOS will query your dhcp server for its IP address then execute the commands issued by the FOG Server.

More to come soon

(place holder)

@Abuelika Interesting… I can say that I have never tested usb booting with muticasting. I can say that the usb booting method was developed for debugging purposes and not for general use. BUT the changes in the Mac firmware put usb booting into higher use.

I think we can make the usb booting to support multicasting if we get your help (since you have the job already setup). I with your multicasting job scheduled on FOG server will you call this url and post the results here?
http://<fog_server_ip>/fog/service/ipxe/boot.php?mac=000000000000

We can use the output here to create a custom usb boot menu item.

Just to add a bit of correlation to your post.

I noticed this when I was working on a POC setup.

On my master fog server the files and folders under /images are owned by root.root with a file mode of 777 (realize I may have done the mode 777 when trying to get the master node to work many months ago. I installed 1.2.0 on this and then upgraded to the latest trunk builds over the months.

On my storage node the replicator created the image files (same as on the master node) being owned by fog.fog with a mode of 755. The storage node was just setup with 1.2.0 and then immediately updated to the latest trunk build.

To answer your question about fog being a sudoer, from a security standpoint I would say no. This should remain a low level account.

@Tom-Elliott The probably the easiest solution is to create a procedure (wiki page) that if you need to update your fog server you need to do:

1. Update the settings in the /opt/fog/.fogsettings
2. Rerun the installer
3. Update the IP address for the storage node on the FOG system where you changed the IP address
4. Update the IP address on a any master storage node that may reference this FOG server
5. Update the FOG_WEB_HOST value
6. update the FOG_TFTP_HOST value

I can I’ll marked this as addressed (Solved), since its not a technical solution but a procedural one that resolves the issue.
[Edit] I would, but it appears I can no longer mark topics as solved[/Edit]

This last weekend I created a new production FOG server and I thought I would document how I moved the images off the root partition and onto a disk of their own. Personally I don’t like storing any dynamic or user data on the root partitions. Too many times in the past I’ve see the root partition fill up and the *nix operating system break hard and the only way to repair them is to rebuild the OS. Putting user files on the root partition is basically the same as for a Windows file server putting the user’s group and home drives on the drive.

For this new production server I created a new VM with 3GB of RAM and a 16GB virtual hard drive. For this project a 16GB virtual hard drive is more than sufficient for the OS and the FOG files. But it isn’t big enough for the typical image files. But we’ll fix that a bit later. While I’m going to cover the process for a virtual machine, the process if very similar for a physical server.

So at this point I have a single VM with 1 vCPU 3GB of RAM and 16GB virtual hard drive. The remainder is the process I used to setup this new production server.

1. Install Centos 7 using the minimal vanilla install for the OS
2. Upgrade the OS with all of the latest fixes by issuing yum upgrade -y from the command prompt
3. Add a new virtual hard drive to the server. Size this new vmdk file appropriately for the number of images you plan on capturing. For my installation I make this new vmdk file 100GB in size.
4. Reboot the server
5. Now we need to locate this new drive. Since this is the second hard drive in the system it would probably be defined as /dev/sdb. Issue the lsblk command to show you the drive structure.

lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 298.1G  0 disk 
├─sda1   8:1    0 294.3G  0 part /
├─sda2   8:2    0     1K  0 part 
└─sda5   8:5    0   3.8G  0 part [SWAP]
sdb      8:16    0 100.0G  0 disk 

Now that we know our 100GB disk is sdb lets connect get down to business prepping the disk for our FOG server.
6) We’ll use LVM for this volume, but lets first create a partition on this disk that fills the entire disk.

fdisk /dev/sdb
n
p
<enter 3 time>
w

7. With the physical partition created lets setup the LVM environment. First we’ll use the pvcreate command initializes physical volume. Remember I only created one partition on the disk so it should be referenced by /dev/sdb1
   pvcreate /dev/sdb1
8. Next we’ll create the volume group. A volume group can be from a single partition to several disks with many partitions. For this example I’ll only attach a single partition to this volume group. But the idea is you can expand your volume group (i.e. add additional storage space to your server) by adding more physical space to this volume group. In the command below I’m creating a volume group name fog_disk and attaching the physical partition /dev/sdb1 to that volume group.
   vgcreate fog_disk /dev/sdb1
9. Now I’m going to create a logical volume that will be linked to the volume group I just created. You can see in the command below I’m creating a logical volume fog_files that will use 100% of the free space on the fog_disk volume group.
   lvcreate -l 100%FREE -n fog_files fog_disk
10. With the logical volume created we can go and format the disk. In this case I’m formatting the disk in the ext4 format. You’ll notice that the device name is the combination of the volume group and the logical volume name.
    mkfs.ext4 /dev/fog_disk/fog_files
11. After formatting the LVM drive we need to attach the new drive to a directory off our root file system. In my situation I want to attach this new disk to the /opt directory. FOG stores some of its files in /opt/fog and the images in /images (off the root partition). When I get done I want to move the /images files to /opt/fog/images so all of FOGs dynamic and user files are on the same physical disk. If we capture more files that our 100GB vmdk drive can hold, it will just fill up that drive. The upload will fail, but it will not take down the OS since the data files are on the /opt partition and not the root partition. Its a little complicated for the MS Windows folks to grasp, but trust me it works this way well. To have this partition connected (mounted) each time the OS boots, we need to append the following to /etc/fstab file. This text needs to be entered on a new line in the fstab file.
    /dev/mapper/fog_disk-fog_files /opt ext4 defaults 0 1
12. With the fstab file updated, lets tell the OS to mount all
    mount -a
13. Issue the df -h command to show you the mounted devices. You should see a line in the df output that looks similar to this:
    /dev/mapper/fog_disk-fog_files 99G 0G 99G 99% /opt
14. With the new hard drive successfully attached to the /opt directory we can go ahead and install FOG as normal.
15. Once FOG is installed we need to do a few clean up steps to get the content of the /images directory into the place I want it.
16. Lets move the entire /images directory into its new location in /opt/fog. Issue the following command
    mv /images /opt/fog
17. Now lets create a new /images folder.
    mkdir /images
18. Now lets ensure that we have things setup correctly before we proceed. If you key in ls -la /images you should not have any files in that directory (hint: we just created it).
19. If you key in this commnad ls -la /opt/fog/images you should see at least 2 directories (/opt/fog/images/dev and /opt/fog/images/postdownloadscripts). If these commands give us what we need then we have everything in place to do the magic of the last step.
20. In this step we are going to bind two directories together. This is kind of like a symbolic link. The issue with a symbolic link here is that nfs can not share a symbolic link. We’ll get around this by using a bind mount to give the appearance that the /images are on the root partition while in reality the files are in the /opt/fog/images directory. Append the following line to /etc/fstab. As with the other fstab line, this text must be on a new line.
    /opt/fog/images /images bind bind 0 0
21. With the fstab updated, issue the following command to mount this bind (link).
    mount -a
22. Great the command completed successfully, how can I tell if it worked? The answer is simple, list the contents of the /images folder. If there is stuff in it, then the command worked. ls -la /images You should see the same content as if you listed the /opt/fog/images directory.

#ls -la /images

total 40
drwxrwxrwx  10 root root 4096 Feb  7 09:55 .
dr-xr-xr-x. 20 root root 4096 Feb  7 09:45 ..
drwxrwxrwx   2 root root 4096 Oct 14 14:45 dev
-rwxrwxrwx   1 root root    0 Feb  5 18:57 .mntcheck
drwxrwxrwx   3 root root 4096 Jan 21 08:54 postdownloadscripts

23. Just to ensure that everything we did survives a boot, reboot the FOG server.
24. Use the df -h command to check to see if the /opt is still mapped to /dev/mapper/fog_disk-fog_files
25. Issue the ls -la /images command to confirm you still have files in that location.
26. Ensure that nfs still has the proper nfs exports created.
    showmount -e 127.0.0.1

Export list for 127.0.0.1:
/images/dev *
/images     *

27. If everything looks like its in place then you are done.

One thing I did not comment on above, by setting up a new vmdk file for the /opt directory, we can expand the vmdk file as needed as our captured images increase. I’ll follow up a bit later on how to add space to your /opt partition with just a few commands.

@Abuelika Sorry, we needed to call the fog server with the multicast job running. The url provided will give us the fog iPXE menu design during a multicast job. I need to collect this to translate it into a grub menu. If you can’t get this I will set this up when I get into the office (in about 2 hrs).

I think we need to get a bit of clarity from the Developers on this one.

On my main node the snapins folder is owned by fog.apache but on my storage node the snapins folder is owned by root.root.

As I posted below on my storage node the /images folder is owned by fog.fog and on my main node /images is owned by root.root.

There doesn’t seem to be any consistency between the user /group and a functioning system (so to speak). But since we are having this discussion something must be amiss here.

@Wayne-Workman Very nice job. I’ll be sure to test this out on both Centos 7 and 6.5 after the holiday. But overall this looks great. It checks all of the boxes on the wish list.

Part 1a

This is more of a proof of concept than as actual desire of mine. Keeping everything in the *nix world is my motto.

With that said. I’ve been able to setup nfs on a windows 2012 R2 server with the following powershell commands.

Import-Module ServerManager
Add-WindowsFeature FS-NFS-Service
Import-Module NFS

mkdir c:\share
mkdir c:\share\images
mkdir c:\share\tftpboot
mkdir c:\share\snapins
mkdir c:\share\snapins\ssl

New-NfsShare –Name "images" –Path c:\share\images –Authentication sys -AllowRootAccess $True -EnableUnmappedAccess $True –Permission Readwrite

Enable-NetFirewallRule -DisplayGroup “Server for NFS” -Verbose

And then on the FOG Master server the following commands will setup the rest of the fog required bits.

mount -t nfs <win_storage-node_ip>:/images /mnt
mkdir /mnt/dev
touch /mnt/.mntcheck
touch /mnt/dev/.mntcheck
umount /mnt

Create a local FTP user for FOG to use.

net localgroup fog_users /add
net user fog_user "mi5ty_cl0ud" /add /EXPIRES:NEVER /PASSWORDCHG:NO /active:YES /Y
net localgroup fog_users fog_user /add
icacls c:\share /grant "fog_users:M"

Setup the FTP server

Install-WindowsFeature Web-FTP-Server,Web-FTP-Service,Web-FTP-Ext -IncludeManagementTools
New-WebFtpSite -Name "FOGFtpSite" -Port 21 -PhysicalPath "c:\share" -IPAddress "<win_storage-node_ip>" 

Set-ItemProperty "IIS:\Sites\FOGFtpSite" -Name ftpServer.security.ssl.controlChannelPolicy -Value 0
Set-ItemProperty "IIS:\Sites\FOGFtpSite" -Name ftpServer.security.ssl.dataChannelPolicy -Value 0
Set-ItemProperty "IIS:\Sites\FOGFtpSite" -Name ftpServer.security.authentication.basicAuthentication.enabled -Value $true
Set-ItemProperty "IIS:\Sites\FOGFtpSite" -Name ftpserver.userisolation.mode -Value 4
Add-WebConfiguration "/system.ftpServer/security/authorization" -value @{accessType="Allow";roles="fog_users";permissions="Read,Write";users=""} -PSPath IIS:\ -location "FOGFtpSite"
Restart-WebItem "IIS:\Sites\FOGFtpSite"

The next part is the web server setup to hand out the FOS kernel and inits. Since I already installed the FTP service which relies on IIS, IIS is already installed. So all we need to do is prep for the FOS files.

Lets first create the directory structure that mimics the path on the FOG master server.

New-Item "IIS:\Sites\Default Web Site\fog" -type Directory
New-Item "IIS:\Sites\Default Web Site\fog\service" -type Directory
New-Item "IIS:\Sites\Default Web Site\fog\service\ipxe" -type Directory

Now that we have the directory, we need to tell IIS to hand out any file that is requested. By default IIS will only pass out files with known extensions like htm, html, asp, and so on. But in our case we want IIS to hand out the inits that end in .xz and the bzImage which doesn’t have an extension. To do this we need to tell IIS to just hand out any file type request (the bzImage is what caused me some pain here, since we are saying any files requested from IIS it will hand out, which could be a security risk if the web site gets hacked)

(Update: 15-May-2017)
You must go manually into IIS management console and add a new mime type of “.*” (dot star without the quotes) and with a type of “application/octet-stream”

There is a better way to configure IIS to send out files with any extension (even no extension) on a per directory basis.

Create the following file: C:\inetpub\wwwroot\fog\service\ipxe\web.config
Insert the following code into that web.config file

<?xml version="1.0" encoding="UTF-8"?>
 <configuration>
     <system.webServer>
         <staticContent>
             <mimeMap fileExtension="." mimeType="application/octet-stream" />
             <mimeMap fileExtension=".*" mimeType="application/octet-stream" />
         </staticContent>
     </system.webServer>
 </configuration>

(End Update: 15-May-2017)

Now that IIS is all setup and ready you will need to copy all of the files from /var/www/html/fog/service/ipxe to the IIS server in the windows path IIS:\Sites\Default Web Site\fog\services\ipxe
You can use the following process to copy the files form your master fog server to the windows storage node.

In an elevated windows command prompt enter
nfsshare fogipxe=C:\inetpub\wwwroot\fog\service\ipxe -o rw sec=sys root unmapped=yes

On your fog server key in:

mount -t nfs <win_storage-node_ip>:/fogipxe /mnt
cp /var/www/html/fog/service/ipxe/* /mnt
umount /mnt

And then finally back on the Windows storage node
nfsshare fogipxe /delete

(Update: 15-May-2017)
The files copied from the fog server seemed to come in with the wrong permissions so we need to reset the permissions on all files from the fog directory and below to the defaults by enabling inheritance on all of the files.
the following command is wrong, its a place holder until the right syntax can be derived

icacls C:\inetpub\wwwroot\fog /grant "fog_users:M"

(End Update: 15-May-2017)

After you’ve copied the files to the correct directory on IIS you should test your setup.
First lets pull the FOG background. Open your browser and key in the url to the background. On my IIS server I’ll use this path, you will need to change the IP address to match your IIS server.
http://<fog_server_ip>/fog/service/ipxe/bg.png
If all goes well you should see the picture of the fog background.

Now lets get a little daring. Lets pull memdisk (a binary file).
http://<fog_server_ip>/fog/service/ipxe/memdisk
If all goes well you should be prompted with a save file dialog.

And then one last test, lets pull a file with an unknown extension.
http://<fog_server_ip>/fog/service/ipxe/refind.efi
Again you should be prompted with a save file dialog.

Onto the next part. For this section we need to install a tftp server to allow pxe booting from your windows storage node. Windows does have a natively built in tftp client, but no tftp server. So for this part we will use an freeware tftp server that I’ve used for years (Tftpd32).

Go to the following URL: http://tftpd32.jounin.net/tftpd32_download.html
Download the tftpd64 service edition (installer)
Launch the installer you just downloaded.
Read and agree to EULA if you accept it continue.
Select (all) Options: Add start menu shortcuts, Add desktop icon, Start service Tftp32_svc, start service monitoring
Use default install location: C:\Program Files\Tftpd64_SE
Tftpd64 Service console should launch
Select the Settings button
Select the GLOBAL tab
Uncheck all options except TFTP Sever. The only selection option we need is “TFTP Server”.
Select the TFTP tab
For the base directory, select the browse button and then navigate to the c:\share\tftpboot folder
Select OK
In the tftp options section enable PXE Compatibility option. Leave all other settings at their default
Press OK
From a command windows with elevated rights

netsh advfirewall firewall add rule name="TFTP Server" dir=in action=allow program="%ProgramFiles%\Tftpd64_SE\tftpd64_svc.exe"
sc stop Tftpd32_svc
sc start Tftpd32_svc

The remaining steps are to copy the contents of the FOG server’s /tftpboot directory to your Windows storage node’s c:\share\tftpboot folder. For this we’ll use NFS to export the directory on the FOG server and then mount that nfs share with your windows server.

In an elevated windows server console key in
nfsshare fogpxe=C:\share\tftpboot -o rw sec=sys root unmapped=yes

On the FOG server

mount -t nfs 192.168.1.205:/fogpxe /mnt
cp -R /tftpboot/* /mnt
umount /mnt

On the Windows storage node from an elevated command prompt
nfsshare fogpxe /delete

If you can make it this far in the setup your storage node should be setup.

Todo list:

1. (done, images are now replicating because Todo #2 was required) Define the Windows storage node in the Master Storage Node (as of the next day all of my images and scripts from the Master storage node are now on the windows storage node)
2. (done, ftp server setup) Determine if the FTP service is really required for a storage node if we will only capture to the master node [edit] duh, ftp server is required for image replication. Must work on this now[/edit]
3. (done, tftp server setup) See if the tftp service is needed if we want to support remote pxe booting [edit] windows 2012 does not have built in tftp server, may have to use tftpd32 or similar if pxe booting is needed [/edit]
4. (www server is setup) Determine if IIS is needed to support serving out the kernel and init images
5. Test this entire setup to see if it actually worls.

@sebastian-roth OK since this is an efi system can we do the trick of,

1. formatting a usb drive as fat32
2. Create a directory structure of /EFI/BOOT
3. Then take /tftpboot/ipxe.efi from the FOG server and place it into that directory structure on the usb flash drive.
4. Rename ipxe.efi in /EFI/BOOT to /EFI/BOOT/BOOTX64.EFI
5. And probably for good measure, take /tftpboot/i386-efi/ipxe.efi from the FOG server and place it into that directory structure on the usb flash drive.
6. Rename ipxe.efi (from the i386-efi directory) in /EFI/BOOT to /EFI/BOOT/BOOTIA32.EFI
7. And finally move that flash drive to a mac and usb boot from it?

If iPXE is happy you should get the FOG iPXE menu.

I can tell you yes, the format needs to be in ldap format and it appears you have the right format as long as the maintenance OU exists under your main OU (like Computers do) then it should place it in the right spot. I can say from one of my installs I have (ou=Desktops,ou=Computers,ou=NYC,ou=US,dc=domain,dc=local)

Since your target computer is ending up in the Computers OU you must have the right information to join the computer to the domain so the FogCrypt part is right too.

When you make a change to the group… everything disappears. That one got me too. FOG applies the information to the host based on the group but the group doesn’t keep the settings. It would be logical for the group to retain this setting but it doesn’t, it is just used to apply the values to the host. I would go into the target host you are interested in and check the AD settings there. Make sure the proper OU settings are there, then redeploy the host again.

@Wayne-Workman You better marry that girl, or I’m going to offer her a job up north (in snowy Michigan) in my group.

I think there is something with the tasks pages. When I just sit on the tasks page the little popup 0 active tasks found is consuming about 8% cpu according to top. On my first test of this that 8% cpu task persisted even when I moved to another page. Then I was getting a session time out popup. Closing the browser and going back in still netted the session timeout. I had to actually log out for that session cookie to reset.

[Edit] After additional testing that 8% httpd process seems to be related to that popup message (i.e. tasks/session/loading/load failed) [/Edit]

(place holder)

I’m not a mac guy, but aren’t they uefi based? If so you are sending the wrong iPXE boot file. UEFI systems need ipxe.efi to boot properly. undionly.kpxe is for bios (legacy) systems.

I’m just pecking in the dark here. But looking at the installer script (actually a called subfunction). This is what is going on in that area.

 add-apt-repository -y ppa:ondrej/php5-5.6 
 if [ "$?" != 0 ]; then
    apt-get update
    apt-get -yq install python-software-properties
    add-apt-repository -y ppa:ondrej/php5-5.6
fi

I might suggest that you try the apt commands one at a time and see what fails. Again I’m just guessing here, these commands will not do anything destructive.

@aruhuno Do you have an old unmanaged switch you can insert between this computer and your building network switch? If Tom is right, inserting a really dumb unmanaged switch should disable the 802.11az port protocol.

@Wayne-Workman you may want to follow this thread.

(place holder)

@Sebastian-Roth said in Mac Mini with T2 chip:

Please try increasing the kernel log level (web UI -> FOG Configuration -> FOG Settings -> FOG Boot Settings -> KERNEL LOGLEVEL) to see if you get any further messages on screen that way.

FWIW: He’ll have to change this in the grub.cfg file on the USB drive. There is a log level entry. I think the default was 4, for more debugging it should be changed to 7

for clarity I found those commands in <fog_trunk>/lib/common/functions.sh In case you want to update your way to an answer.

@aruhuno The devs worked on the init scripts over the weekend. Since you just upgraded to the current trunk release, does the network interface come up when you try to register this Lenovo today?

If the updates inits do not work. Could you repeat the same steps from earlier with this modification?

Use this udhcpc command
udhcpc -i eth0 --now

Then when the command times out (or better picks up an up address)
cat /sys/class/net/eth0/carrier

If the carrier value is still 0 wait 30 seconds and issue the udhcpc command again. What they were looking at is to find a way to detect if the network adapter has a slow startup. They are working hard to find a way to get these network adapters to init right.