@Sebastian-Roth I’ve added the graphing. Below is a graph generated with just 4 test entries in the DB. I’ve changed all the graphs to display 30 instead of 20, and made the images a bit bigger.

@Sebastian-Roth thanks for merging.

I’ve got the information storing pieces worked out in the server side via this PR. Next is to get the graphs created.

@sebastian-roth

https://github.com/FOGProject/fogproject/pull/471
https://github.com/FOGProject/fogproject/pull/470

New Script

#!/bin/bash

# Get the OS Information.
read -r os_name os_version <<< $(lsb_release -ir | cut -d':' -f2 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | tr '\n' ' ')

# Get the FOG Version.
source /opt/fog/.fogsettings
system_class_php=${docroot}/${webroot}/lib/fog/system.class.php
fog_version=$(cat ${system_class_php} | grep FOG_VERSION | cut -d',' -f2 | cut -d"'" -f2)

# Construct correct mysql options.
options="-sN"
if [[ $snmysqlhost != "" ]]; then
        options="$options -h$snmysqlhost"
fi
if [[ $snmysqluser != "" ]]; then
        options="$options -u$snmysqluser"
fi
if [[ $snmysqlpass != "" ]]; then
        options="$options -p$snmysqlpass"
fi
options="$options -D $mysqldbname -e"

# Construct sql statements.
FOG_TFTP_PXE_KERNEL_32_select='select settingValue from globalSettings WHERE settingKey = "FOG_TFTP_PXE_KERNEL_32";'
FOG_TFTP_PXE_KERNEL_select='select settingValue from globalSettings WHERE settingKey = "FOG_TFTP_PXE_KERNEL";'
FOG_TFTP_PXE_KERNEL_DIR_select='select settingValue from globalSettings WHERE settingKey = "FOG_TFTP_PXE_KERNEL_DIR";'
FOG_HOSTS_KERNELS_select='SELECT UNIQUE hostKernel FROM hosts;'

# Execute sql statements, get values.
FOG_TFTP_PXE_KERNEL_32=$(mysql $options "$FOG_TFTP_PXE_KERNEL_32_select")
FOG_TFTP_PXE_KERNEL=$(mysql $options "$FOG_TFTP_PXE_KERNEL_select")
FOG_TFTP_PXE_KERNEL_DIR=$(mysql $options "$FOG_TFTP_PXE_KERNEL_DIR_select")
FOG_HOST_KERNELS=$(mysql $options "$FOG_HOSTS_KERNELS_select")

# Get kernel information.
## Begin building the JSON array to send.
kernel_versions_info='['

# Begin processing global 32 bit kernel.
# Check if 32 bit global kernel file exists.
if [[ -f ${FOG_TFTP_PXE_KERNEL_DIR}${FOG_TFTP_PXE_KERNEL_32} ]]; then
    # Get file information.
    file_information=$(file --no-pad --brief ${FOG_TFTP_PXE_KERNEL_DIR}${FOG_TFTP_PXE_KERNEL_32})
    # Check if this is a linux kernel or not.
    if [[ "${file_information}" == *"Linux kernel"* ]]; then
        # Here, we are pretty sure the current file we're looking at is a Linux kernel. Parse the version information.
        version=$(echo ${file_information} | cut -d, -f2 | sed 's/version*//' | xargs)
        # If there are any double quotes in this version information, add a backslash in front of them for JSON escaping.
        version=$(echo $version | sed 's/"/\\"/g')
	# Prepend the filename to the version.
	version="${FOG_TFTP_PXE_KERNEL_32} ${version}"
	# Wrap the version in double quotes for JSON syntax.
        version="\"${version}\""
        # Check if the last character in the kernel_versions_info variable is a double quote. If so, add a leading comma.
        if [[ "${kernel_versions_info: -1}" == '"' ]]; then
            version=",${version}"
        fi
        # Append version to kernel_versions_info JSON list.
        kernel_versions_info="${kernel_versions_info}${version}"
    fi
fi

# Begin processing 64 bit global kernel.
# Check if global kernel file exists.
if [[ -f ${FOG_TFTP_PXE_KERNEL_DIR}${FOG_TFTP_PXE_KERNEL} ]]; then
    # Get file information.
    file_information=$(file --no-pad --brief ${FOG_TFTP_PXE_KERNEL_DIR}${FOG_TFTP_PXE_KERNEL})
    # Check if this is a linux kernel or not.
    if [[ "${file_information}" == *"Linux kernel"* ]]; then
        # Here, we are pretty sure the current file we're looking at is a Linux kernel. Parse the version information.
        version=$(echo ${file_information} | cut -d, -f2 | sed 's/version*//' | xargs)
        # If there are any double quotes in this version information, add a backslash in front of them for JSON escaping.
        version=$(echo $version | sed 's/"/\\"/g')
        # Prepend the filename to the version.
        version="${FOG_TFTP_PXE_KERNEL} ${version}"
        # Wrap the version in double quotes for JSON syntax.
        version="\"${version}\""
        # Check if the last character in the kernel_versions_info variable is a double quote. If so, add a leading comma.
        if [[ "${kernel_versions_info: -1}" == '"' ]]; then
            version=",${version}"
        fi
        # Append version to kernel_versions_info JSON list.
        kernel_versions_info="${kernel_versions_info}${version}"
    fi
fi

# Begin processing each unique host kernel that is not a global kernel.
for host_kernel in $FOG_HOST_KERNELS; do
    # Check if this is the name of the 32 or 64 bit global kernel. If so, skip it.
    if [[ "${host_kernel}" != "${FOG_TFTP_PXE_KERNEL}" && "${host_kernel}" != "${FOG_TFTP_PXE_KERNEL_32}" ]]; then
        if [[ -f ${FOG_TFTP_PXE_KERNEL_DIR}${host_kernel} ]]; then
            # Get file information.
            file_information=$(file --no-pad --brief ${FOG_TFTP_PXE_KERNEL_DIR}${host_kernel})
            # Check if this is a linux kernel or not.
            if [[ "${file_information}" == *"Linux kernel"* ]]; then
                # Here, we are pretty sure the current file we're looking at is a Linux kernel. Parse the version information.
                version=$(echo ${file_information} | cut -d, -f2 | sed 's/version*//' | xargs)
                # If there are any double quotes in this version information, add a backslash in front of them for JSON escaping.
                version=$(echo $version | sed 's/"/\\"/g')
                # Prepend the filename to the version.
                version="${host_kernel} ${version}"
                # Wrap the version in double quotes for JSON syntax.
                version="\"${version}\""
                # Check if the last character in the kernel_versions_info variable is a double quote. If so, add a leading comma.
                if [[ "${kernel_versions_info: -1}" == '"' ]]; then
                    version=",${version}"
                fi
                # Append version to kernel_versions_info JSON list.
                kernel_versions_info="${kernel_versions_info}${version}"
            fi
        fi
    fi    
done

# Finish JSON list formatting.
kernel_versions_info="${kernel_versions_info}]"

# Format payload.
payload='{"fog_version":"'${fog_version}'","os_name":"'${os_name}'","os_version":"'${os_version}'","kernel_versions_info":'${kernel_versions_info}'}'

#echo "os_name=${os_name}"
#echo "os_version=${os_version}"
#echo "fog_version=${fog_version}"
#echo "kernel_versions_info=${kernel_versions_info}"
#echo "payload=${payload}"

# Send to reporting endpoint.
curl -s -X POST -H "Content-Type: application/json" -d "${payload}" https://fog-external-reporting-entries.theworkmans.us:/api/records

New Payload

{
  "fog_version": "1.5.9.139",
  "os_name": "Debian",
  "os_version": "11",
  "kernel_versions_info": [
    "bzImage32 5.15.19 (buildkite-agent@Tollana) #1 SMP Thu Feb 3 15:05:47 CST 2022",
    "bzImage 5.15.19 (buildkite-agent@Tollana) #1 SMP Thu Feb 3 15:10:05 CST 2022",
    "arm_Image_test little-endian",
    "another_test_kernel 4.19.145 (sebastian@Tollana) #1 SMP Sun Sep 13 05:43:10 CDT 2020"
  ]
}

Notes

The script got lengthy, I think it’s fine though. I prepended the filename to the version information. I noticed the FOG ARM kernels lack version information - even when not parsed. This is it unparsed:

Linux kernel ARM64 boot executable Image, little-endian, 4K pages

If version information is added to future ARM kernels, the script should pick it up. I did add three hosts to my test fog system. Two of them I assigned valid kernel files to, and the third one I assigned a bogus file name. The script checks to make sure all the kernels actually exist, and are actually linux kernels before adding them to the payload.

@fry_p said in Windows 11/Future for Us:

Just out of curiosity, how much would it cost to have Microsoft to sign the certs?

Microsoft should consider doing this for FOG as a donation.

@sebastian-roth Good points. Let me refine this a little with the DB query stuff.

I’ve opened two PRs for this.
https://github.com/FOGProject/fogproject/pull/469
https://github.com/FOGProject/fogproject/pull/468

This is the new reporting script that gets kernel information.

Proposed reporting script

#!/bin/bash


# Get the OS Information.
read -r os_name os_version <<< $(lsb_release -ir | cut -d':' -f2 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | tr '\n' ' ')


# Get the FOG Version.
source /opt/fog/.fogsettings
system_class_php=${docroot}/${webroot}/lib/fog/system.class.php
fog_version=$(cat ${system_class_php} | grep FOG_VERSION | cut -d',' -f2 | cut -d"'" -f2)


# Get kernel information.
## Begin building the JSON array to send.
kernel_versions_info='['
## Loop over all files where fog kernels are normally stored.
for filename in ${docroot}/${webroot}/service/ipxe/*; do
    # Get the absolute paths of each file, for cleaner handling.
    absolute_path=$(readlink -f ${filename})
    # Get file information about each file.
    file_information=$(file --no-pad --brief $absolute_path)
    # Check if "Linux kernel" is a substring in the file information or not.
    if [[ "${file_information}" == *"Linux kernel"* ]]; then
        # Here, we are pretty sure the current file we're looking at is a Linux kernel. Parse the version information.
        version=$(echo ${file_information} | cut -d, -f2 | sed 's/version*//' | xargs)
        # If there are any double quotes in this version information, add a backslash in front of them for JSON escaping.
        version=$(echo $version | sed 's/"/\\"/g')
        # Wrap the version in double quotes for JSON syntax.
        version="\"${version}\""
        # Check if the last character in the kernel_versions_info variable is a double quote. If so, add a leading comma.
        if [[ "${kernel_versions_info: -1}" == '"' ]]; then
            version=",${version}"
        fi
        # Append version to kernel_versions_info JSON list.
        kernel_versions_info="${kernel_versions_info}${version}"
    fi
done
# Finish JSON list formatting.
kernel_versions_info="${kernel_versions_info}]"


# Format payload.
payload='{"fog_version":"'${fog_version}'","os_name":"'${os_name}'","os_version":"'${os_version}'","kernel_versions_info":'${kernel_versions_info}'}'



#echo "os_name=${os_name}"
#echo "os_version=${os_version}"
#echo "fog_version=${fog_version}"
#echo "kernel_versions_info=${kernel_versions_info}"
#echo "payload=${payload}"

# Send to reporting endpoint.
curl -s -X POST -H "Content-Type: application/json" -d "${payload}" https://fog-external-reporting-entries.theworkmans.us:/api/records

Payload

This is sample output from my freshly installed Debian 11 testbox, using dev-branch as of today. This is the JSON that would get sent to the reporting API.

{
    "fog_version":"1.5.9.139",
    "os_name":"Debian",
    "os_version":"11",
    "kernel_versions_info":[
        "5.15.19 (buildkite-agent@Tollana) #1 SMP Thu Feb 3 15:10:05 CST 2022",
        "5.15.19 (buildkite-agent@Tollana) #1 SMP Thu Feb 3 15:05:47 CST 2022",
        "2.6.13.1 (mdv@localhost) #1 Tue Sep 13 18:18:41 CST 2005",
        "MEMDISK 3.86 2010-04-01"
    ]
}

How I plan to incorporate the kernel info

I’ll need to add another table to the database (no big deal) with an autonumber column, datetime column, and a version varchar column of 255 length. Any version info over 255 characters would just get chopped. FOG’s kernel version info string is about 68 characters, fyi.

With those three columns, I’ll be able to count the occurrences of each unique kernel version within the last 7 days, and produce graphs to report that much like the other graphs already being made.

Note 1

A single fog server can have many kernels. The above script gets information for ALL linux kernels in the kernel directory including the one for grub and memdisk which I think is probably a good thing. To get the exact kernels in use via global settings we’d need to do one of these:

**A) query the API which involves tokens (I don’t think this is best)
**B) look into the database to see what’s in use (much easier)

The script to query the DB to get the absolute paths to the default kernels in use would look something like this, plus some parsing & formatting.

source /opt/fog/.fogsettings # This line is already in the reporting script.

# Construct correct mysql options.
options="-sN"
if [[ $snmysqlhost != "" ]]; then
        options="$options -h$snmysqlhost"
fi
if [[ $snmysqluser != "" ]]; then
        options="$options -u$snmysqluser"
fi
if [[ $snmysqlpass != "" ]]; then
        options="$options -p$snmysqlpass"
fi
options="$options -D fog -e"

# Construct sql statement.
statement='select settingKey, settingValue from globalSettings WHERE settingKey = "FOG_TFTP_PXE_KERNEL_32" or settingKey = "FOG_TFTP_PXE_KERNEL" or settingKey = "FOG_TFTP_PXE_KERNEL_DIR";'

# Execute the query.
mysql $options "$statement"

Output would typically be:

FOG_TFTP_PXE_KERNEL     bzImage
FOG_TFTP_PXE_KERNEL_32  bzImage32
FOG_TFTP_PXE_KERNEL_DIR /var/www/fog//service/ipxe/

So the question here from Note 1 is “do we want to get only the globally configured kernels and nothing else, or do we want to get all kernel information?” I’d think skipping all this and just getting all of it would be best. Thoughts?

Note 2

I decided to purposely not include the kernel file name, as this may contain some sort of identifiable information. Also I purposely chopped some of the file command output, to get only version information and nothing else.

Note 3

grub & memdisk kernels are really old. 2005 and 2010. What’s involved with updating these?

user pings

@developers @moderators @testers

@Sebastian-Roth as well as all the others ( @developers @moderators @testers )

Would we want anything besides kernel information? I had some ideas:

Number of hosts
Number of images
Number of each type of image
Number of systems imaged in the last 7 days
Number of storage nodes
Number of storage groups
Other things?

It’s been said before, but will say it again. This is all anonymous data. There is no way to correlate this data back to any system.

It just provides us insight (and possibly bragging rights). imagine us being able to say “FOG imaged a bazillion systems last week, we’re proud of that” or maybe “FOG assists with managing a quadrillion million computer systems every day”. I’m being funny with the imaginary figures here, but you get the idea. It’d be nice to make statements like that on the home page fogproject.org

George’s post is valid. Just wanted to add that regular ole storage nodes within the same storage group will load share. You’d set the max hosts on each storage node to accomplish this.

Hey there, I noticed the installation tests for Ubuntu 18 and 20 hit the hard limits I have set for installation on working-1.6 branch. For the other branches this is around 2 and 3 minutes. The hard-limit I have set is 15 minutes.

Both are hanging on this step:

* Installing package: mariadb-server..........................

The error logs suggest the package repositories for Ubuntu 18 and 20 were having problems at the time. I’ll keep an eye on it tomorrow to see if it repeats.

fwiw, I have the 15-minute hard limit set to avoid cost. If something hangs / breaks and never completes, I don’t want AWS instances staying on for days (or weeks) because I didn’t notice it. So the 15 minute hard-limit prevents that, cutting the processes off at that time and shutting down the instance for the next test, or to be done for the day.

@sebastian-roth I’ll look into it. I assume you mean FOS kernel, not FOG server’s kernel. Hold on the release while I poke at this please.

I see Fedora 35 tests are now passing on dev-branch as well as working-1.6.

He’s probably looking at master branch.

For the record, dev-branch is kept more updated, but it’s not an official release. Though a lot of people use it (see usage report in my signature).

At least he didn’t say 5 years sourceforge hasn’t been updated in 5 years. Interestingly the traffic graph says it gets 50 weekly downloads.
https://sourceforge.net/projects/freeghost/files/FOG/stats/timeline

@p4cm4n Is your org able to help fog with resources? Be they time or financial? FOG is running critically low these things, and support is the one thing that comes to mind when you talk about 70 nodes and 14,000 clients.

@sebastian-roth Would you like dev-branch-php8 included in the daily tests?

You might look over the wiki articles on the topic.
https://wiki.fogproject.org/wiki/index.php?title=Snapin_Examples
https://wiki.fogproject.org/wiki/index.php?title=SnapinPacks

“Fault Tolerance” is a huge, generic term. I hear it so often in my job, that it doesn’t mean anything to me anymore.

What is your written & agreed upon RPO?
What is your written & agreed upon RTO?
What is your written & agreed upon SLA?
https://www.acronis.com/en-us/articles/rto-rpo/
https://wow24-7.io/blog/what-is-sla-service-level-agreements-and-their-role-in-business

This is completely possible. For spreading things out, any of the following is possible.

Multiple FOG Masters to separate FOG Client traffic / node traffic from your traffic / engineer traffic.

• This solves the nervous user problem (clicking buttons in the UI over and over rapidly).
• This solves load issues with 15 engineers / technicians just sitting on the dashboard with the graphs chewing up server CPU.
• 1 FOG Master for people to interact with.
• 1 FOG Master for everything else (node check-ins, FOG Client check-ins)
• Could split this out even further if you wanted. Could setup load balancing with F5 or HAProxy, for example. Or, could do simple DNS load balancing by creating many records of the same DNS name, but each pointing to different FOG Server IP Addresses.

One or more seperate storage nodes for imaging load

• Build separate Storage Nodes, as many as you think you need. Set one as the storage master of your storage group. Only storage group masters can multicast.
• An image can be assigned to multiple storage groups. So you could have two storage nodes, each a master of a separate storage group. One will be the image master (the original), and that master will replicate to the other master.
• Replication documentation here.
• Multi-casting documentation here.
• Multi-casting troubleshooting documentation here.

If your geographically dispersed, use the location plugin.

• Documentation can be found here. This covers the topic with an older FOG version but is still accurate.

Put Database on dedicated VM.

• Setup MariaDB on another VM. Set password, remote access. Documentation here.
• Dump your database from your fog server, load to new DB.
• For all your your fog masters and storage nodes: Point them to the new database via this file: /opt/fog/.fogsettings. Documentation here.
• Re-run fog installers on all fog masters and all storage nodes.

Increase Client Checkin Time

• This will resolve a lot of the client load.
• Documentation here.

@sebastian-roth said in Fedora 35 & FOG:

Though there is at least one and I kind of feel obliged to at least see how much work it is to fix this.

There’s some Fedora 25 users out there. Maybe I set those up for the school district in 2016