PXE-less booting FOS client OS

[george1421]

Now that we have the capabilities to query FOG database variables during the ( (b)ash ) post install scripts [ https://forums.fogproject.org/topic/6463/expose-fog-host-and-image-properties-to-post-install-scripts ], it would be interesting to see if its possible to replace PXE booting for select (troubled) hardware with USB bootable FOS image.

The main issue I see is in implementing such a system is that for pxe booting kernel boot time parameters are passed from the iPXE boot menu to the bzImage as its is launched. For USB booting there would not be any intelligent boot parameters because these are currently handled programically by the boot.php script. So the USB FOG client would need a different way to collect these kernel parameters. Enter in the feature request listed above. Using a similar method the USB booting FOS client could query the FOG server to request this information.

For the USB booting FOS client we will use GRUB as the boot menu managers, to give a similar experience as the FOG iPXE menu. The GRUB boot loader could pass a kernel parameter to bzImage (and then onto to the inits) to signal it being a USB boot. This flag would then instruct the FOS client to query for the missing kernel parameters (or just eliminate the kernel parameters all together in favor of the php query method.

I’ve looked in the inits and these are the only scripts I found (so far) that reference the kernel parameters directly.
/bin/fog.capone
/bin/fog.checkin
/bin/fog.sysinfo
/usr/share/fog/lib/funcs.sh

With funcs.sh actually doing the conversion from kernel parameters to (b)ash variables. It looks like it would be trivial (so to speak) to check if the USB boot kernel parameter existed and then launch the direct query to the fog server which would populate the required (b)ash variables.

I understand that some concessions would have to be made because the grub boot would be pretty much static. The USB boot FOS client would only support the basics of registration, image capture and image deploy. Any other functionality beyond that would be nice to haves.

We already have guidance on how to create the FOS usb boot drive [ https://forums.fogproject.org/topic/6532/usb-boot-target-device-into-fog-os-live-fosl-for-debugging ] so we do have the majority of the required technology already developed. We would just have to develop a php page that would create the required kernel parameters as exported (b)ash variables as we have with the hostinfo.php script.

[Tom Elliott]

If I pass all the kernel arguments (besides isdebug, debug, and normal kernel args) All I’d have to do for the scripts that run is source the files after calling the hostinfo.php script.

[Tom Elliott]

I’ve setup the hostinfo.php to assign the imaging kernel args information now as well.

I will try to work on getting this so “registered” hosts will no longer need this information passed in the kernel args. The only one that would be needed would be the falseTasking (which is used for non-imaged hosts) as we don’t have a record of what is to be happening.

[george1421]

@Tom-Elliott Just thinking about this (using hostinfo.php) will any bits of that code fall down if the host is unregistered and that page is called. Thinking for a new host that never was seen by fog, how will that code react? The sql query (of my prototype code) would just not return any records. I do know how the updated code will react if there is no host previously defined. Would another page (kerninfo.php) be a better choice?

BUT having hostinfo.php post host AND kernel info opens some interesting opportunities. This means that host info would be immediately available to all scripts running in FOS (including all post install scripts). The IT admin developing post install scripts would not have to do anything extra to post these variables.

[Tom Elliott]

@george1421 I think less “let me take over” is to simply check if the variables are already set. If they are not set, use what is in the hostinfo.php file.

Basically instead of just doing:

export <somevariable>='<somevalue>'

We should do something like:
[[ -z $<somevariable> ]] && <somevariable>='<somevalue>'

[Tom Elliott]

I’ve added the suggested code so if the variable is already set, not to set it, otherwise set it up as needed.

[george1421]

@Tom-Elliott My thought was (must not have explained it very well since I don’t understand completely the logic you used), is just thinking ahead. What happens if the system has not been previously registered as in a new system. Will the host lookup fail (abend) or just return null? I can test this out tonight. I’ll spin up a new vm and see if I can break the hostinfo.php. I just don’t want something unexpected to impact another part of the code. But I think what has done so far is great and will let FOG move is a few directions to fill some gaps. So thank you for the attention on this.

[george1421]

This is only for reference so I don’t forget where I’m at

Ok this is a rough outline of what I need to add to maybe to the /bin/fog or functs.sh script this will need to be rolled into the init.zx images when tested and approved.

 if [ "${boottype}" = "usb" ]; then
    iface=`ip addr | grep 'MULTICAST,UP,LOWER_UP' | awk '{print $2}' | cut -d: -f 1`;
    ipaddr=$(ip addr show ${iface} | awk '/inet / {print $2}' | cut -d/ -f 1);
    mac=$(ip link show ${iface} | awk '/ether/ {print $2}');
    wget -q -O /tmp/hinfo.txt "http://${web-url}/service/hostinfo.php?mac=${mac}";
    if [ -f "/tmp/hinfo.txt" ]; then
      . /tmp/hinfo.txt
      rm -f /tmp/hinfo.txt;
    fi;
  fi

[edit] something in the above code was messing up the code block. [/edit]
In this case I’m introducing a new kernel parameter called boottype if its set to usb then it will run the hostinfo.php code to query the fog server for the required kernel parameters that were not passed because we booted off a usb stick and not iPXE.

As it stands right now I have all of the parts, by themselves work (in concept). Now I just need to glue the bits together and test it on my hardware.

[george1421]

The first run through (well third try) in debug mode was successful.

My test image is based on the work that was done for FOSL usb boot drive.
https://forums.fogproject.org/topic/6532/usb-boot-target-device-into-fog-os-live-fosl-for-debugging

for reference here is the grub.conf

set timeout=100
set default=0
insmod all_video

menuentry "Local hard drive" {
    rootnoverify (hd0,1)
    chainloader +1
}

menuentry "FOG Image Deploy" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 boottype=usb consoleblank=0 web=192.168.1.188/fog/ rootfstype=ext4 type=down isdebug=yes 
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Run Memtest86+" {
 linux /boot/memdisk iso raw
 initrd /boot/memtest.bin
}

menuentry "Perform Full Host Registration and Inventory" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=manreg
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Quick Registration and Inventory" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=autoreg
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "FOG Image Capture" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 boottype=usb consoleblank=0 web=192.168.188/fog/ rootfstype=ext4 type=up isdebug=yes
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Client System Information (Compatibility)" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=sysinfo
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "FOG 64-bit Debug Kernel" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 keymap= boottype=usb consoleblank=0 isdebug=yes
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

The next part invovles extracting the current init.xz file and the updating the funcs.sh file.
The following code was inserted into the /usr/share/fog/lib/funcs.sh file just after the section that parses out the kernel parameters.

if [ "${boottype}" = "usb" ] && [ "${web}" != "" ]; then
  iface=`ip addr | grep 'MULTICAST,UP,LOWER_UP' | awk '{print $2}' | cut -d: -f 1`;
  ipaddr=$(ip addr show ${iface} | awk '/inet / {print $2}' | cut -d/ -f 1);
  mac=$(ip link show ${iface} | awk '/ether/ {print $2}');
  if [ "${mac}" != "" ]; then
    wget -q -O /tmp/hinfo.txt "$http://{web}service/hostinfo.php?mac=${mac}";
    if [ -f "/tmp/hinfo.txt" ]; then
      . /tmp/hinfo.txt
      rm -f /tmp/hinfo.txt;
    fi;
  fi;
fi

Then recompress the init and move it to the FOSL usb boot drive.

The completed test was as follows:

1. Schedule a image deployment for the target computer via the FOG web gui
2. Boot from the FOSL and select Image Deploy from the GRUB menu.
3. Single step through the deployment process
4. Success!!

There is much more debugging that is required. I’m concerned about the code to detect the mac address. I don’t know what happens if there is multiple network adapters in the target computer? What happens if there is multiple network adapters and both have a link up message? How does a wifi adapter image the mac address selection (I do make some assumptions with the code).

With that said, I feel confident that I’m on the right path to make a PXE-less FOG deployment/capture reality.

[Tom Elliott]

@george1421 I’ve edited the code to read as:

### If USB Boot device we need a way to get the kernel args properly
if [[ $boottype == usb && ! -z $web ]]; then
    iface=$(ip addr | grep 'MULTICAST,UP,LOWER_UP' | awk '{print $2}' | cut -d: -f 1)
    ipaddr=$(ip addr show $iface | awk '/inet / {print $2}' | cut -d/ -f 1)
    mac=$(ip link show $iface | awk '/ether/ {print $2}')
    if [[ ! -z $mac ]]; then
        wget -q -O /tmp/hinfo.txt "http://${web}service/hostinfo.php?mac=$mac"
        if [[ -f /tmp/hinfo.txt ]]; then
            . /tmp/hinfo.txt
            rm -f /tmp/hinfo.txt
        fi
    fi
fi

The kernels I build have NO wifi address, and while (rarely) they show up, they’re not connected to anything until you tell them to connect to something. I think it’s safe to say it won’t have a link up for WIFI addresses.

I have a function already in the funcs.sh called getMACAddresses, we also have verifyNetworkConnection. Is it really a need to get the interface information at that point? Meaning, the native scripts already go through much of this checking. Your code in the funcs needs a validation area too I suppose, but all scripts that load the funcs.sh will consistently be checking the same information over and over. Maybe have this portion added to bin/fog before the first if statement? Then the only check we need is after the kernel load in. We wouldn’t need the interface checking as it’s already been done. Just my thoughts.

Where is the ‘$web’ defined in the USB? Is that done at load time that remains static?

Maybe we should have error checking in the code base after the initial check just to inform the user if there’s a problem?

Here’s, now that some time has passed, what I’ve done:

In /bin/fog below the `. /usr/share/fog/lib/funcs.sh I have added:

### If USB Boot device we need a way to get the kernel args properly
if [[ $boottype == usb && ! -z $web ]]; then
    mac=$(getMACAddresses)
    wget -q -O /tmp/hinfo.txt "http://${web}service/hostinfo.php?mac=$mac"
    [[ -f /tmp/hinfo.txt ]] && . /tmp/hinfo.txt
fi

In /usr/share/fog/lib/funcs.sh I’ve added after the kernel reload up bits,

### If USB Boot device we need a way to get the kernel args properly
[[ $boottype == usb && -f /tmp/hinfo.txt ]] && . /tmp/hinfo.txt

This way we don’t have to continuously run polls to the server as we step through each of the scripts. It’s much more simplified I think this way too.

[george1421]

Just an update on this topic.

Tom made several enhancements to my proposed code and its now (silently) pushed in the latest trunk version.

As of now, this is the grub configuration file

set timeout=20
set default=0
insmod all_video

menuentry "Local hard drive" {
    rootnoverify (hd0,1)
    chainloader +1
}

menuentry "FOG Image Capture/Deploy" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 boottype=usb consoleblank=0 web=192.168.1.188/fog/ rootfstype=ext4
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Perform Full Host Registration and Inventory" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=manreg
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Quick Registration and Inventory" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=autoreg
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Client System Information (Compatibility)" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=sysinfo
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Run Memtest86+" {
 linux /boot/memdisk iso raw
 initrd /boot/memtest.bin
}

menuentry "FOG 64-bit Debug Kernel" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 keymap= boottype=usb consoleblank=0 isdebug=yes
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

There still needs to be some refinement in the FOG init scripts. One error I created is when I booted the USB FOS client in deploy mode without scheduling a deploy task in the fog webgui first. This caused one of the FOG scripts to get confused and essentially blow up. I went back and scheduled an image deploy task then rebooted the USB FOS client and the image deployed correctly. So I need to look into the scripts to see if I can trap this condition (IT tech tells the FOS client to deploy without a scheduled task to deploy waiting). As the project goes this solution is very close to being reality.

[george1421]

Just as a recommendation to properly trap the condition of a IT tech USB booting the FOS client into a capture/deploy mode without the first scheduling a capture/deploy task via the FOG Management GUI, I would ask the following concept added to tail code in /bin/fog

This says that if the FOS Client was booted from USB and the type is Null (as in the case of no action waiting for the target host) trap the error and inform the IT Tech what action to take next. The wise guy in me would probably say “Nothing to do here, go away” but that might not translate to well globally.

    case $type in
        down)
            fog.download
            ;;
        up)
            fog.upload
            ;;
        *)
            [[ -z $type ]] && type="Null"
                #New code added here
                if [ $boottype == usb ]; then
                   handleError "Fatal Error: No scheduled task was found for this host.\n\nPlease schedule a task using the FOG webgui before usb booting this target host.\n"
                else
                    handleError "Fatal Error: Unknown request type :: $type"
                fi;
            ;;
    esac

[ITSolutions]

@george1421 Maybe give the option “If no scheduled task to complete, would you like to do a quick image?” I see that is not an option on the GRUB menu, not sure if that would be a viable option to build into this. But it would be helpful at times.

[george1421]

@ITSolutions The issue I see it (being able to quick image) as two fold.

1. The target computer is already in the FOS OS, when that error is thrown, with a certain set of kernel parameters, to load quick image it would have to chain load to itself to change the kernel parameters (I don’t know if that is possible)
2. I looked into the quick image but ran into an issue. The quick image is actually 2 iPXE screens. One is the main menu where quick image menu entry is and the second allows you to pick an image for deployment. With GRUB I can’t do dynamic screens (i.e update a grub menu based on a database query for the available images).

I wouldn’t say the issue is impossible, but right now I want to focus on getting the basic image to capture and deploy from a USB boot. At this time I’m almost ready to call it complete.

I can run all of the menu items and they work ( even reregistering a host, which reset all of the host parameters (not sure if that is a good thing) ). That all passes without error. The only one I’m stumbling on right now is booting to the local hard drive, and I may drop that if I can’t get it to work on more than a few systems.

[ITSolutions]

@george1421 Sounds great, like I had said I wasn’t sure if it was possible for quick image. If not I don’t see it as a big deal in my opinion. The exit to the HDD I think is a very small issue, If you want to go to the HDD just remove the USB key. lol
I have been hoping for a usb boot option for a while, but I have very limit knowledge in creating such a thing and not enough time to explore and learn. Glad you and Tom are able to work on this part.

[Tom Elliott]

This is really easy to add.

Remember, we allow a method of creating a new host that asks if you would like to image the computer?

Just a little extra code, but we can build the requests right into a script.

[george1421]

@Tom-Elliott Tom you have to remember you can do everything. My programming tool box, is a bit limited.

If I can see and read the code I can copy it, that’s about it for programming.

[Tom Elliott]

Ask and you will have an example:
This is straight from the “do you want to image now” but it should give a basic flow logic base.

 askme=""
 while [[ -z $askme ]]; do
     echo -n "    Would you like to image this computer now? (y/N) "
     read askme
     case $askme in
         [Nn]|[Nn][Oo]|"")
             askme="N"
             ;;  
         [Yy]|[Yy][Ee][Ss])
             tmp=""
             ret=""
             retry=3
             while [[ $ret != "#!ok" && ! $retry -eq 0 ]]; do
                 echo " * Enter FOG GUI Username and Password"
                 echo -n "    Username: "
                 read username
                 echo -n "    Password: "
                 read -s password
                 user64=$(echo $username | tr -d '\012' | base64)
                 pass64=$(echo $password | tr -d '\012' | base64)
                 ret=$(wget --post-data="mac=$mac&username=$user64&password=$pass64" -qO - http://  ${web}service/checkcredentials.php 2>/dev/null)
                 case $ret in
                     '#!ok')
                         echo " * This host will reboot and imaging will start!"
                         ret=$tmp
                         realdoimage=$(echo -n 1 | base64)
                         break
                         ;;  
                     '#!il')
                         echo " * Error: Invalid Login! ($retry remaining)"
                         let retry-=1
                         ;;  
                 esac
             done
             askme="Y"
             ;;  
         *)  
             askme=""
             echo " * Invalid input, please try again"
             ;;  
     esac
 done

[george1421]

@Tom-Elliott That would be inserted in this section then?

                if [ $boottype == usb ]; then
                   handleError "Fatal Error: No scheduled task was found for this host.\n\nPlease schedule a task using the FOG webgui before usb booting this target host.\n"
                else

Instead of just telling them to make sure they setup a task, it would prompt them to run a quick image?

[edit] Thinking about this more when you ask to image the machine the FOS client reboots and then boot.php set the right kernel parameters to start imaging. Is there a way to do this without rebooting? Like running the /bin/fog script again?

[edit2] Just for clarity I don’t see anything where that script posts the do imaging now is that what realdoimage does?

[Tom Elliott]

@george1421 Again this was just to show an example of an approach.

Technically, with the use of your hostinfo.php stuff we no longer need to require a secondary reboot as the hostinfo.php will have the relevant information already accessible (with the exception that the registration base64 encodes all the variables where newer items no longer need this.)

It’s just an example I didn’t take anything for it though.