PXE-less booting FOS client OS

[george1421]

@Tom-Elliott My thought was (must not have explained it very well since I don’t understand completely the logic you used), is just thinking ahead. What happens if the system has not been previously registered as in a new system. Will the host lookup fail (abend) or just return null? I can test this out tonight. I’ll spin up a new vm and see if I can break the hostinfo.php. I just don’t want something unexpected to impact another part of the code. But I think what has done so far is great and will let FOG move is a few directions to fill some gaps. So thank you for the attention on this.

[george1421]

This is only for reference so I don’t forget where I’m at

Ok this is a rough outline of what I need to add to maybe to the /bin/fog or functs.sh script this will need to be rolled into the init.zx images when tested and approved.

 if [ "${boottype}" = "usb" ]; then
    iface=`ip addr | grep 'MULTICAST,UP,LOWER_UP' | awk '{print $2}' | cut -d: -f 1`;
    ipaddr=$(ip addr show ${iface} | awk '/inet / {print $2}' | cut -d/ -f 1);
    mac=$(ip link show ${iface} | awk '/ether/ {print $2}');
    wget -q -O /tmp/hinfo.txt "http://${web-url}/service/hostinfo.php?mac=${mac}";
    if [ -f "/tmp/hinfo.txt" ]; then
      . /tmp/hinfo.txt
      rm -f /tmp/hinfo.txt;
    fi;
  fi

[edit] something in the above code was messing up the code block. [/edit]
In this case I’m introducing a new kernel parameter called boottype if its set to usb then it will run the hostinfo.php code to query the fog server for the required kernel parameters that were not passed because we booted off a usb stick and not iPXE.

As it stands right now I have all of the parts, by themselves work (in concept). Now I just need to glue the bits together and test it on my hardware.

[george1421]

The first run through (well third try) in debug mode was successful.

My test image is based on the work that was done for FOSL usb boot drive.
https://forums.fogproject.org/topic/6532/usb-boot-target-device-into-fog-os-live-fosl-for-debugging

for reference here is the grub.conf

set timeout=100
set default=0
insmod all_video

menuentry "Local hard drive" {
    rootnoverify (hd0,1)
    chainloader +1
}

menuentry "FOG Image Deploy" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 boottype=usb consoleblank=0 web=192.168.1.188/fog/ rootfstype=ext4 type=down isdebug=yes 
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Run Memtest86+" {
 linux /boot/memdisk iso raw
 initrd /boot/memtest.bin
}

menuentry "Perform Full Host Registration and Inventory" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=manreg
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Quick Registration and Inventory" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=autoreg
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "FOG Image Capture" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 boottype=usb consoleblank=0 web=192.168.188/fog/ rootfstype=ext4 type=up isdebug=yes
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Client System Information (Compatibility)" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=sysinfo
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "FOG 64-bit Debug Kernel" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 keymap= boottype=usb consoleblank=0 isdebug=yes
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

The next part invovles extracting the current init.xz file and the updating the funcs.sh file.
The following code was inserted into the /usr/share/fog/lib/funcs.sh file just after the section that parses out the kernel parameters.

if [ "${boottype}" = "usb" ] && [ "${web}" != "" ]; then
  iface=`ip addr | grep 'MULTICAST,UP,LOWER_UP' | awk '{print $2}' | cut -d: -f 1`;
  ipaddr=$(ip addr show ${iface} | awk '/inet / {print $2}' | cut -d/ -f 1);
  mac=$(ip link show ${iface} | awk '/ether/ {print $2}');
  if [ "${mac}" != "" ]; then
    wget -q -O /tmp/hinfo.txt "$http://{web}service/hostinfo.php?mac=${mac}";
    if [ -f "/tmp/hinfo.txt" ]; then
      . /tmp/hinfo.txt
      rm -f /tmp/hinfo.txt;
    fi;
  fi;
fi

Then recompress the init and move it to the FOSL usb boot drive.

The completed test was as follows:

1. Schedule a image deployment for the target computer via the FOG web gui
2. Boot from the FOSL and select Image Deploy from the GRUB menu.
3. Single step through the deployment process
4. Success!!

There is much more debugging that is required. I’m concerned about the code to detect the mac address. I don’t know what happens if there is multiple network adapters in the target computer? What happens if there is multiple network adapters and both have a link up message? How does a wifi adapter image the mac address selection (I do make some assumptions with the code).

With that said, I feel confident that I’m on the right path to make a PXE-less FOG deployment/capture reality.

[Tom Elliott]

@george1421 I’ve edited the code to read as:

### If USB Boot device we need a way to get the kernel args properly
if [[ $boottype == usb && ! -z $web ]]; then
    iface=$(ip addr | grep 'MULTICAST,UP,LOWER_UP' | awk '{print $2}' | cut -d: -f 1)
    ipaddr=$(ip addr show $iface | awk '/inet / {print $2}' | cut -d/ -f 1)
    mac=$(ip link show $iface | awk '/ether/ {print $2}')
    if [[ ! -z $mac ]]; then
        wget -q -O /tmp/hinfo.txt "http://${web}service/hostinfo.php?mac=$mac"
        if [[ -f /tmp/hinfo.txt ]]; then
            . /tmp/hinfo.txt
            rm -f /tmp/hinfo.txt
        fi
    fi
fi

The kernels I build have NO wifi address, and while (rarely) they show up, they’re not connected to anything until you tell them to connect to something. I think it’s safe to say it won’t have a link up for WIFI addresses.

I have a function already in the funcs.sh called getMACAddresses, we also have verifyNetworkConnection. Is it really a need to get the interface information at that point? Meaning, the native scripts already go through much of this checking. Your code in the funcs needs a validation area too I suppose, but all scripts that load the funcs.sh will consistently be checking the same information over and over. Maybe have this portion added to bin/fog before the first if statement? Then the only check we need is after the kernel load in. We wouldn’t need the interface checking as it’s already been done. Just my thoughts.

Where is the ‘$web’ defined in the USB? Is that done at load time that remains static?

Maybe we should have error checking in the code base after the initial check just to inform the user if there’s a problem?

Here’s, now that some time has passed, what I’ve done:

In /bin/fog below the `. /usr/share/fog/lib/funcs.sh I have added:

### If USB Boot device we need a way to get the kernel args properly
if [[ $boottype == usb && ! -z $web ]]; then
    mac=$(getMACAddresses)
    wget -q -O /tmp/hinfo.txt "http://${web}service/hostinfo.php?mac=$mac"
    [[ -f /tmp/hinfo.txt ]] && . /tmp/hinfo.txt
fi

In /usr/share/fog/lib/funcs.sh I’ve added after the kernel reload up bits,

### If USB Boot device we need a way to get the kernel args properly
[[ $boottype == usb && -f /tmp/hinfo.txt ]] && . /tmp/hinfo.txt

This way we don’t have to continuously run polls to the server as we step through each of the scripts. It’s much more simplified I think this way too.

[george1421]

Just an update on this topic.

Tom made several enhancements to my proposed code and its now (silently) pushed in the latest trunk version.

As of now, this is the grub configuration file

set timeout=20
set default=0
insmod all_video

menuentry "Local hard drive" {
    rootnoverify (hd0,1)
    chainloader +1
}

menuentry "FOG Image Capture/Deploy" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 boottype=usb consoleblank=0 web=192.168.1.188/fog/ rootfstype=ext4
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Perform Full Host Registration and Inventory" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=manreg
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Quick Registration and Inventory" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=autoreg
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Client System Information (Compatibility)" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=4 initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=192.168.1.188/fog/ boottype=usb consoleblank=0 rootfstype=ext4 loglevel=4 mode=sysinfo
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

menuentry "Run Memtest86+" {
 linux /boot/memdisk iso raw
 initrd /boot/memtest.bin
}

menuentry "FOG 64-bit Debug Kernel" {
 echo loading the kernel
 linux  /boot/bzImage loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 keymap= boottype=usb consoleblank=0 isdebug=yes
 echo loading the virtual hard drive
 initrd /boot/init.xz
 echo booting kernel...
}

There still needs to be some refinement in the FOG init scripts. One error I created is when I booted the USB FOS client in deploy mode without scheduling a deploy task in the fog webgui first. This caused one of the FOG scripts to get confused and essentially blow up. I went back and scheduled an image deploy task then rebooted the USB FOS client and the image deployed correctly. So I need to look into the scripts to see if I can trap this condition (IT tech tells the FOS client to deploy without a scheduled task to deploy waiting). As the project goes this solution is very close to being reality.

[george1421]

Just as a recommendation to properly trap the condition of a IT tech USB booting the FOS client into a capture/deploy mode without the first scheduling a capture/deploy task via the FOG Management GUI, I would ask the following concept added to tail code in /bin/fog

This says that if the FOS Client was booted from USB and the type is Null (as in the case of no action waiting for the target host) trap the error and inform the IT Tech what action to take next. The wise guy in me would probably say “Nothing to do here, go away” but that might not translate to well globally.

    case $type in
        down)
            fog.download
            ;;
        up)
            fog.upload
            ;;
        *)
            [[ -z $type ]] && type="Null"
                #New code added here
                if [ $boottype == usb ]; then
                   handleError "Fatal Error: No scheduled task was found for this host.\n\nPlease schedule a task using the FOG webgui before usb booting this target host.\n"
                else
                    handleError "Fatal Error: Unknown request type :: $type"
                fi;
            ;;
    esac

[ITSolutions]

@george1421 Maybe give the option “If no scheduled task to complete, would you like to do a quick image?” I see that is not an option on the GRUB menu, not sure if that would be a viable option to build into this. But it would be helpful at times.

[george1421]

@ITSolutions The issue I see it (being able to quick image) as two fold.

1. The target computer is already in the FOS OS, when that error is thrown, with a certain set of kernel parameters, to load quick image it would have to chain load to itself to change the kernel parameters (I don’t know if that is possible)
2. I looked into the quick image but ran into an issue. The quick image is actually 2 iPXE screens. One is the main menu where quick image menu entry is and the second allows you to pick an image for deployment. With GRUB I can’t do dynamic screens (i.e update a grub menu based on a database query for the available images).

I wouldn’t say the issue is impossible, but right now I want to focus on getting the basic image to capture and deploy from a USB boot. At this time I’m almost ready to call it complete.

I can run all of the menu items and they work ( even reregistering a host, which reset all of the host parameters (not sure if that is a good thing) ). That all passes without error. The only one I’m stumbling on right now is booting to the local hard drive, and I may drop that if I can’t get it to work on more than a few systems.

[ITSolutions]

@george1421 Sounds great, like I had said I wasn’t sure if it was possible for quick image. If not I don’t see it as a big deal in my opinion. The exit to the HDD I think is a very small issue, If you want to go to the HDD just remove the USB key. lol
I have been hoping for a usb boot option for a while, but I have very limit knowledge in creating such a thing and not enough time to explore and learn. Glad you and Tom are able to work on this part.

[Tom Elliott]

This is really easy to add.

Remember, we allow a method of creating a new host that asks if you would like to image the computer?

Just a little extra code, but we can build the requests right into a script.

[george1421]

@Tom-Elliott Tom you have to remember you can do everything. My programming tool box, is a bit limited.

If I can see and read the code I can copy it, that’s about it for programming.

[Tom Elliott]

Ask and you will have an example:
This is straight from the “do you want to image now” but it should give a basic flow logic base.

 askme=""
 while [[ -z $askme ]]; do
     echo -n "    Would you like to image this computer now? (y/N) "
     read askme
     case $askme in
         [Nn]|[Nn][Oo]|"")
             askme="N"
             ;;  
         [Yy]|[Yy][Ee][Ss])
             tmp=""
             ret=""
             retry=3
             while [[ $ret != "#!ok" && ! $retry -eq 0 ]]; do
                 echo " * Enter FOG GUI Username and Password"
                 echo -n "    Username: "
                 read username
                 echo -n "    Password: "
                 read -s password
                 user64=$(echo $username | tr -d '\012' | base64)
                 pass64=$(echo $password | tr -d '\012' | base64)
                 ret=$(wget --post-data="mac=$mac&username=$user64&password=$pass64" -qO - http://  ${web}service/checkcredentials.php 2>/dev/null)
                 case $ret in
                     '#!ok')
                         echo " * This host will reboot and imaging will start!"
                         ret=$tmp
                         realdoimage=$(echo -n 1 | base64)
                         break
                         ;;  
                     '#!il')
                         echo " * Error: Invalid Login! ($retry remaining)"
                         let retry-=1
                         ;;  
                 esac
             done
             askme="Y"
             ;;  
         *)  
             askme=""
             echo " * Invalid input, please try again"
             ;;  
     esac
 done

[george1421]

@Tom-Elliott That would be inserted in this section then?

                if [ $boottype == usb ]; then
                   handleError "Fatal Error: No scheduled task was found for this host.\n\nPlease schedule a task using the FOG webgui before usb booting this target host.\n"
                else

Instead of just telling them to make sure they setup a task, it would prompt them to run a quick image?

[edit] Thinking about this more when you ask to image the machine the FOS client reboots and then boot.php set the right kernel parameters to start imaging. Is there a way to do this without rebooting? Like running the /bin/fog script again?

[edit2] Just for clarity I don’t see anything where that script posts the do imaging now is that what realdoimage does?

[Tom Elliott]

@george1421 Again this was just to show an example of an approach.

Technically, with the use of your hostinfo.php stuff we no longer need to require a secondary reboot as the hostinfo.php will have the relevant information already accessible (with the exception that the registration base64 encodes all the variables where newer items no longer need this.)

It’s just an example I didn’t take anything for it though.

[george1421]

@Tom-Elliott But it is very close to what we need. We would just have to post to the fog server that we wanted to image, refresh the hostinfo.php and then relaunch the main fog script. It sounds plausible.

The part 2 of this is I can see is someone will want to then (next) quick image an unregistered computer like they can do today with the iPXE menus. That wouldn’t be such an easy task since we would have to then query for available images (inside FOS) update the variables and then launch the quick image function.

[edit] Now that I think about it, except for the query image to deploy, the rest is possible without much headache.

[Tom Elliott]

@george1421 It’s not that difficult. In fact, quick image is probably even simpler.

[george1421]

For this part of the PXE-less boot the FOS client, I think its a wrap (finished). There is sure some nice to haves that can be done inside the inits but for the usb and the usb booting part I think I’ve reached the end. While I’ve only tested this on a bios system, I expect it to work under uefi too.

Here is the final grub menu for this iteration. I plan on writing a complete tutorial on how to build this usb later tonight.

set myfogip=192.168.1.88
set myimage=/boot/bzImage
set myinits=/boot/init.xz
set myloglevel=4
set timeout=-1
insmod all_video

menuentry "1. FOG Image Deploy/Capture" {
 echo loading the kernel
 linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

menuentry "2. Perform Full Host Registration and Inventory" {
 echo loading the kernel
 linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=manreg
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

menuentry "3. Quick Registration and Inventory" {
 echo loading the kernel
 linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=autoreg
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

menuentry "4. Client System Information (Compatibility)" {
 echo loading the kernel
 linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=127000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=sysinfo
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

menuentry "5. Run Memtest86+" {
 linux /boot/memdisk iso raw
 initrd /boot/memtest.bin
}

menuentry "6. FOG Debug Kernel" {
 echo loading the kernel
 linux  $myimage loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=127000 keymap= boottype=usb consoleblank=0 rootfstype=ext4 isdebug=yes
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

[Sebastian Roth]

Great stuff you’re coming up with here @george1421! Unfortunately I didn’t have the time to read it all to the last bit. So forgive me if I am asking things you already wrote about here. How are you going to put this all together to boot from USB/ISO? Are you using grub-mkrescue as mentioned here? Just curious…

[george1421]

@Sebastian-Roth No I will (probably) just modify my bash script I used to create the FOS-L usb. That seemed to be the easiest and creates both bios and uefi usb images on the same usb. I need to review that tutorial I did and then just merge in the new GRUB boot menu. Really there is no magic that goes on in the flash drive other than the grub boot menu and a few tweaks Tom did to some files in the inits, well and the hostinfo.php from another feature request. The difficult part was just gluing all of the bits together.

[george1421]

@Tom-Elliott Just thinking about the next steps here…

I have to work on a few extra projects for my real job right now, but I’m thinking ahead a little.

In funcs.sh, can we move the code to pickup the kernel parameters and to call hostinfo.php into its own sub function? That we can reuse and call at different points in both the init scripts and via the post install scripts? That way I can reuse that code and have a consistent way of getting the parameters.

I could imagine a sub called getKernelParams() that could be called, and in that sub function if the ( boottype=usb ) then it would call a second function getHostInfo() to pick up the host info (that way too we can call getHostInfo() by its self to pick up the host details from the post install script(s) too (at this time I personally don’t care about the kernel parameters in the post install script, but if I did I could just call the getKernelParams instead). Then of course the /bin/fog script would need to take advantage of these new functions.