[Feedback Requested] Adding kernel info to FOG Reporting

Wayne Workman

This is the new reporting script that gets kernel information.

Proposed reporting script

#!/bin/bash


# Get the OS Information.
read -r os_name os_version <<< $(lsb_release -ir | cut -d':' -f2 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | tr '\n' ' ')


# Get the FOG Version.
source /opt/fog/.fogsettings
system_class_php=${docroot}/${webroot}/lib/fog/system.class.php
fog_version=$(cat ${system_class_php} | grep FOG_VERSION | cut -d',' -f2 | cut -d"'" -f2)


# Get kernel information.
## Begin building the JSON array to send.
kernel_versions_info='['
## Loop over all files where fog kernels are normally stored.
for filename in ${docroot}/${webroot}/service/ipxe/*; do
    # Get the absolute paths of each file, for cleaner handling.
    absolute_path=$(readlink -f ${filename})
    # Get file information about each file.
    file_information=$(file --no-pad --brief $absolute_path)
    # Check if "Linux kernel" is a substring in the file information or not.
    if [[ "${file_information}" == *"Linux kernel"* ]]; then
        # Here, we are pretty sure the current file we're looking at is a Linux kernel. Parse the version information.
        version=$(echo ${file_information} | cut -d, -f2 | sed 's/version*//' | xargs)
        # If there are any double quotes in this version information, add a backslash in front of them for JSON escaping.
        version=$(echo $version | sed 's/"/\\"/g')
        # Wrap the version in double quotes for JSON syntax.
        version="\"${version}\""
        # Check if the last character in the kernel_versions_info variable is a double quote. If so, add a leading comma.
        if [[ "${kernel_versions_info: -1}" == '"' ]]; then
            version=",${version}"
        fi
        # Append version to kernel_versions_info JSON list.
        kernel_versions_info="${kernel_versions_info}${version}"
    fi
done
# Finish JSON list formatting.
kernel_versions_info="${kernel_versions_info}]"


# Format payload.
payload='{"fog_version":"'${fog_version}'","os_name":"'${os_name}'","os_version":"'${os_version}'","kernel_versions_info":'${kernel_versions_info}'}'



#echo "os_name=${os_name}"
#echo "os_version=${os_version}"
#echo "fog_version=${fog_version}"
#echo "kernel_versions_info=${kernel_versions_info}"
#echo "payload=${payload}"

# Send to reporting endpoint.
curl -s -X POST -H "Content-Type: application/json" -d "${payload}" https://fog-external-reporting-entries.theworkmans.us:/api/records

Payload

This is sample output from my freshly installed Debian 11 testbox, using dev-branch as of today. This is the JSON that would get sent to the reporting API.

{
    "fog_version":"1.5.9.139",
    "os_name":"Debian",
    "os_version":"11",
    "kernel_versions_info":[
        "5.15.19 (buildkite-agent@Tollana) #1 SMP Thu Feb 3 15:10:05 CST 2022",
        "5.15.19 (buildkite-agent@Tollana) #1 SMP Thu Feb 3 15:05:47 CST 2022",
        "2.6.13.1 (mdv@localhost) #1 Tue Sep 13 18:18:41 CST 2005",
        "MEMDISK 3.86 2010-04-01"
    ]
}

How I plan to incorporate the kernel info

I’ll need to add another table to the database (no big deal) with an autonumber column, datetime column, and a version varchar column of 255 length. Any version info over 255 characters would just get chopped. FOG’s kernel version info string is about 68 characters, fyi.

With those three columns, I’ll be able to count the occurrences of each unique kernel version within the last 7 days, and produce graphs to report that much like the other graphs already being made.

Note 1

A single fog server can have many kernels. The above script gets information for ALL linux kernels in the kernel directory including the one for grub and memdisk which I think is probably a good thing. To get the exact kernels in use via global settings we’d need to do one of these:

**A) query the API which involves tokens (I don’t think this is best)
**B) look into the database to see what’s in use (much easier)

The script to query the DB to get the absolute paths to the default kernels in use would look something like this, plus some parsing & formatting.

source /opt/fog/.fogsettings # This line is already in the reporting script.

# Construct correct mysql options.
options="-sN"
if [[ $snmysqlhost != "" ]]; then
        options="$options -h$snmysqlhost"
fi
if [[ $snmysqluser != "" ]]; then
        options="$options -u$snmysqluser"
fi
if [[ $snmysqlpass != "" ]]; then
        options="$options -p$snmysqlpass"
fi
options="$options -D fog -e"

# Construct sql statement.
statement='select settingKey, settingValue from globalSettings WHERE settingKey = "FOG_TFTP_PXE_KERNEL_32" or settingKey = "FOG_TFTP_PXE_KERNEL" or settingKey = "FOG_TFTP_PXE_KERNEL_DIR";'

# Execute the query.
mysql $options "$statement"

Output would typically be:

FOG_TFTP_PXE_KERNEL     bzImage
FOG_TFTP_PXE_KERNEL_32  bzImage32
FOG_TFTP_PXE_KERNEL_DIR /var/www/fog//service/ipxe/

So the question here from Note 1 is “do we want to get only the globally configured kernels and nothing else, or do we want to get all kernel information?” I’d think skipping all this and just getting all of it would be best. Thoughts?

Note 2

I decided to purposely not include the kernel file name, as this may contain some sort of identifiable information. Also I purposely chopped some of the file command output, to get only version information and nothing else.

Note 3

grub & memdisk kernels are really old. 2005 and 2010. What’s involved with updating these?

user pings

@developers @moderators @testers

Wayne Workman

I’ve opened two PRs for this.
https://github.com/FOGProject/fogproject/pull/469
https://github.com/FOGProject/fogproject/pull/468

Sebastian Roth

@Wayne-Workman Great stuff, thanks for this. Pretty busy at the moment but I will reply in more detail over the weekend.

Sebastian Roth

@wayne-workman said in [Feedback Requested] Adding kernel info to FOG Reporting:

A single fog server can have many kernels.

On the one hand side it would be great to see more than just the active kernel binary (global settings) but on the other hand we can’t tell if people really use all those kernels or just have those sitting there unused. From my point of view it would get us the most clear picture about kernels in use if we check the database for global kernel filename (64 & 32 bit) and possibly also kernels assigned in host settings (just another simple query to the hosts table. Open for discussion. What do you all think?

I decided to purposely not include the kernel file name, as this may contain some sort of identifiable information. Also I purposely chopped some of the file command output, to get only version information and nothing else.

While I can see that filenames might be a particular name chosen by the admin I don’t think this would reveal too much valuable information. I just say this because I was looking for a way to distinguish between bzImage (64 bit) and bzImage32 (32 bit) using Linux commands but have not been able to yet. Using filenames would help. On the other hand the script could just check the filename and add 32, 64 bit or ARM64 information according to the filename.

If we really use the information from the database we don’t need to think about this part too much.

grub & memdisk kernels are really old. 2005 and 2010. What’s involved with updating these?

I don’t really know where those binaries came from in the first place. Those were added before I joined. Keeping things up tp date is usually a good thing but it can break stuff as well. Some people have complained about us updating rEFInd a couple of times and I am not sure if we should touch these (grub & memdisk). As well I am not sure if newer versions exist. As far as I know this is a grub4dos binary - no update since 2009. Probably similar with memdisk?!?

Wayne Workman

@sebastian-roth Good points. Let me refine this a little with the DB query stuff.

Wayne Workman

New Script

#!/bin/bash

# Get the OS Information.
read -r os_name os_version <<< $(lsb_release -ir | cut -d':' -f2 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | tr '\n' ' ')

# Get the FOG Version.
source /opt/fog/.fogsettings
system_class_php=${docroot}/${webroot}/lib/fog/system.class.php
fog_version=$(cat ${system_class_php} | grep FOG_VERSION | cut -d',' -f2 | cut -d"'" -f2)

# Construct correct mysql options.
options="-sN"
if [[ $snmysqlhost != "" ]]; then
        options="$options -h$snmysqlhost"
fi
if [[ $snmysqluser != "" ]]; then
        options="$options -u$snmysqluser"
fi
if [[ $snmysqlpass != "" ]]; then
        options="$options -p$snmysqlpass"
fi
options="$options -D $mysqldbname -e"

# Construct sql statements.
FOG_TFTP_PXE_KERNEL_32_select='select settingValue from globalSettings WHERE settingKey = "FOG_TFTP_PXE_KERNEL_32";'
FOG_TFTP_PXE_KERNEL_select='select settingValue from globalSettings WHERE settingKey = "FOG_TFTP_PXE_KERNEL";'
FOG_TFTP_PXE_KERNEL_DIR_select='select settingValue from globalSettings WHERE settingKey = "FOG_TFTP_PXE_KERNEL_DIR";'
FOG_HOSTS_KERNELS_select='SELECT UNIQUE hostKernel FROM hosts;'

# Execute sql statements, get values.
FOG_TFTP_PXE_KERNEL_32=$(mysql $options "$FOG_TFTP_PXE_KERNEL_32_select")
FOG_TFTP_PXE_KERNEL=$(mysql $options "$FOG_TFTP_PXE_KERNEL_select")
FOG_TFTP_PXE_KERNEL_DIR=$(mysql $options "$FOG_TFTP_PXE_KERNEL_DIR_select")
FOG_HOST_KERNELS=$(mysql $options "$FOG_HOSTS_KERNELS_select")

# Get kernel information.
## Begin building the JSON array to send.
kernel_versions_info='['

# Begin processing global 32 bit kernel.
# Check if 32 bit global kernel file exists.
if [[ -f ${FOG_TFTP_PXE_KERNEL_DIR}${FOG_TFTP_PXE_KERNEL_32} ]]; then
    # Get file information.
    file_information=$(file --no-pad --brief ${FOG_TFTP_PXE_KERNEL_DIR}${FOG_TFTP_PXE_KERNEL_32})
    # Check if this is a linux kernel or not.
    if [[ "${file_information}" == *"Linux kernel"* ]]; then
        # Here, we are pretty sure the current file we're looking at is a Linux kernel. Parse the version information.
        version=$(echo ${file_information} | cut -d, -f2 | sed 's/version*//' | xargs)
        # If there are any double quotes in this version information, add a backslash in front of them for JSON escaping.
        version=$(echo $version | sed 's/"/\\"/g')
	# Prepend the filename to the version.
	version="${FOG_TFTP_PXE_KERNEL_32} ${version}"
	# Wrap the version in double quotes for JSON syntax.
        version="\"${version}\""
        # Check if the last character in the kernel_versions_info variable is a double quote. If so, add a leading comma.
        if [[ "${kernel_versions_info: -1}" == '"' ]]; then
            version=",${version}"
        fi
        # Append version to kernel_versions_info JSON list.
        kernel_versions_info="${kernel_versions_info}${version}"
    fi
fi

# Begin processing 64 bit global kernel.
# Check if global kernel file exists.
if [[ -f ${FOG_TFTP_PXE_KERNEL_DIR}${FOG_TFTP_PXE_KERNEL} ]]; then
    # Get file information.
    file_information=$(file --no-pad --brief ${FOG_TFTP_PXE_KERNEL_DIR}${FOG_TFTP_PXE_KERNEL})
    # Check if this is a linux kernel or not.
    if [[ "${file_information}" == *"Linux kernel"* ]]; then
        # Here, we are pretty sure the current file we're looking at is a Linux kernel. Parse the version information.
        version=$(echo ${file_information} | cut -d, -f2 | sed 's/version*//' | xargs)
        # If there are any double quotes in this version information, add a backslash in front of them for JSON escaping.
        version=$(echo $version | sed 's/"/\\"/g')
        # Prepend the filename to the version.
        version="${FOG_TFTP_PXE_KERNEL} ${version}"
        # Wrap the version in double quotes for JSON syntax.
        version="\"${version}\""
        # Check if the last character in the kernel_versions_info variable is a double quote. If so, add a leading comma.
        if [[ "${kernel_versions_info: -1}" == '"' ]]; then
            version=",${version}"
        fi
        # Append version to kernel_versions_info JSON list.
        kernel_versions_info="${kernel_versions_info}${version}"
    fi
fi

# Begin processing each unique host kernel that is not a global kernel.
for host_kernel in $FOG_HOST_KERNELS; do
    # Check if this is the name of the 32 or 64 bit global kernel. If so, skip it.
    if [[ "${host_kernel}" != "${FOG_TFTP_PXE_KERNEL}" && "${host_kernel}" != "${FOG_TFTP_PXE_KERNEL_32}" ]]; then
        if [[ -f ${FOG_TFTP_PXE_KERNEL_DIR}${host_kernel} ]]; then
            # Get file information.
            file_information=$(file --no-pad --brief ${FOG_TFTP_PXE_KERNEL_DIR}${host_kernel})
            # Check if this is a linux kernel or not.
            if [[ "${file_information}" == *"Linux kernel"* ]]; then
                # Here, we are pretty sure the current file we're looking at is a Linux kernel. Parse the version information.
                version=$(echo ${file_information} | cut -d, -f2 | sed 's/version*//' | xargs)
                # If there are any double quotes in this version information, add a backslash in front of them for JSON escaping.
                version=$(echo $version | sed 's/"/\\"/g')
                # Prepend the filename to the version.
                version="${host_kernel} ${version}"
                # Wrap the version in double quotes for JSON syntax.
                version="\"${version}\""
                # Check if the last character in the kernel_versions_info variable is a double quote. If so, add a leading comma.
                if [[ "${kernel_versions_info: -1}" == '"' ]]; then
                    version=",${version}"
                fi
                # Append version to kernel_versions_info JSON list.
                kernel_versions_info="${kernel_versions_info}${version}"
            fi
        fi
    fi    
done

# Finish JSON list formatting.
kernel_versions_info="${kernel_versions_info}]"

# Format payload.
payload='{"fog_version":"'${fog_version}'","os_name":"'${os_name}'","os_version":"'${os_version}'","kernel_versions_info":'${kernel_versions_info}'}'

#echo "os_name=${os_name}"
#echo "os_version=${os_version}"
#echo "fog_version=${fog_version}"
#echo "kernel_versions_info=${kernel_versions_info}"
#echo "payload=${payload}"

# Send to reporting endpoint.
curl -s -X POST -H "Content-Type: application/json" -d "${payload}" https://fog-external-reporting-entries.theworkmans.us:/api/records

New Payload

{
  "fog_version": "1.5.9.139",
  "os_name": "Debian",
  "os_version": "11",
  "kernel_versions_info": [
    "bzImage32 5.15.19 (buildkite-agent@Tollana) #1 SMP Thu Feb 3 15:05:47 CST 2022",
    "bzImage 5.15.19 (buildkite-agent@Tollana) #1 SMP Thu Feb 3 15:10:05 CST 2022",
    "arm_Image_test little-endian",
    "another_test_kernel 4.19.145 (sebastian@Tollana) #1 SMP Sun Sep 13 05:43:10 CDT 2020"
  ]
}

Notes

The script got lengthy, I think it’s fine though. I prepended the filename to the version information. I noticed the FOG ARM kernels lack version information - even when not parsed. This is it unparsed:

Linux kernel ARM64 boot executable Image, little-endian, 4K pages

If version information is added to future ARM kernels, the script should pick it up. I did add three hosts to my test fog system. Two of them I assigned valid kernel files to, and the third one I assigned a bogus file name. The script checks to make sure all the kernels actually exist, and are actually linux kernels before adding them to the payload.

Sebastian Roth

@Wayne-Workman Awesome! Tested on a test VM with Ubuntu as well. Works like a charm.

If version information is added to future ARM kernels, the script should pick it up. I did add three hosts to my test fog system. Two of them I assigned valid kernel files to, and the third one I assigned a bogus file name. The script checks to make sure all the kernels actually exist, and are actually linux kernels before adding them to the payload.

Would you mind opening pull requests for dev-branch and working-1.6 again?

Wayne Workman

@sebastian-roth

https://github.com/FOGProject/fogproject/pull/471
https://github.com/FOGProject/fogproject/pull/470

Wayne Workman

@Sebastian-Roth thanks for merging.

I’ve got the information storing pieces worked out in the server side via this PR. Next is to get the graphs created.

Wayne Workman

@Sebastian-Roth I’ve added the graphing. Below is a graph generated with just 4 test entries in the DB. I’ve changed all the graphs to display 30 instead of 20, and made the images a bit bigger.

Wayne Workman

Putting a bow-tie on this thread: This feature is complete. Link to the external reporting is in my signature, the kernels in use are beginning to show up, these figures will increase and show a bigger picture over time.