Popularity Contest

[Tom Elliott]

@Sebastian-Roth, @Wayne-Workman, @george1421 I agree.

I think the GUID generation can be done relatively simply. We may already have a GUID generated at install time. (FOG_UUID in FOG Settings I think.) Though I don’t know how generalized it is, and it’s more a random ID, not an actual UUID (in UUID Standard)

So my thoughts:

Keep this “polling/analytics” separate from the main fog code base (Maybe under utils?)

Ask, on next update/install, if the admin would like to send analytics data to fog, and potentially what to send?

Store this section to it’s own configuration file.

So the asking requests something like:

Would you like to send analytic data for FOG? (y/N)
We would like to collect the information once a week, on Sunday’s at 03:00:00. (This allows us to delete entries older than a week.)

The information we would like to collect are:

• Randomly Generated UUID -> This allows us to track the information anonymously, but also not continually add the request to the queue, data is inserted once, then updated as needed.
• FOG Version -> This will allow us to track what versions of FOG are currently in use.
• OS -> This tells us what type of OS you are using to host the fog server.
• OS Version -> This tells us what version of OS you are using to host the fog server
• Timestamp -> Just the date and time this was sent.

We allow the user to select what they’re comfortable sending. I think this is just a better method overall.

The utility should simply create a simple shell script to send the data the user would like in real time. To collect the FOG Version, should should be able to do a lookup of the /var/www/fog/lib/fog/system.class.php and look up FOG_VERSION in the file.
OS and OS Version should look at lsb_release -a if the command exists, or look at /etc/os-release, or fallback as needed. Finally, the utility would then create the crontab to perform the task regularly.

Getting UUID = echo $(cat /proc/sys/kernel/random/uuid)
We call the configuration file something like /opt/foganalytics/.foganalytics
The script would be called something like /opt/foganalytics/foganalytics
Crontab would look like: 0 3 * * 0 /usr/bin/bash /opt/foganalytics/foganalytics >/dev/null 2&>1

Does this all make sense and sound doable?

[Wayne Workman]

Great approach Tom. If a crontab is setup to only send once per week, the unique identifier becomes totally unnecessary. Because as you suggested, we can delete the collected data on Friday, and have all new fresh data by Monday - or keep all the data, and build reports based on the last week only.

One issue I see to this approach: having potentially many thousand fog servers all reporting in at the same time I think isn’t going to work out. I’d suggest the minute value of the Crontab be randomized between 0 and 59. This will spread the load out for the receiving API.

[Wayne Workman]

@developers what branch would you like the PR to be created to?

[Junkhacker]

@Wayne-Workman i realize i’m coming into this really late, but why not randomize what day of the week we collect the data as well, and we can just look at the window of “last 7 days” when we look at the data?

[Wayne Workman]

@Junkhacker I think this is the right move.

[Wayne Workman]

I have the information collection portion of the server completed. Code is currently sitting in my fork of fog-community-scripts.

Here’s what the request looks like right now:

curl -X POST -H "Content-Type: application/json"  -d '{"fog_version":"12.34.56","os_name":"Debian","os_version":"10"}' http://fog-popularity-contest.theworkmans.us:/api/records

^ That’s a live link, should work for whoever tries it. If it’s stored ok, the reply is

{"message":"record recorded"}

Here’s a screenshot of the first test record saved, and the table layout.

I’ll work on the client side next, will follow the conclusion of the guidance provided by everyone here.

I do plan on adding https to the server side.
As far as presentation of the data, not sure on that yet. Kicking around ideas of rendering some basic graphs in .png format every day and serving those, as well as making a daily database dump available for download.

[Junkhacker]

@Wayne-Workman shall we make the stats public, because “why not” and to be transparent about what we collect?

[Wayne Workman]

@Junkhacker absolutely. In fact, I was planning to make daily dumps of the database available.

[Sebastian Roth]

@george1421 Sorry I have not found enough time to look into this last week. Would have been nice to add this to the upcoming FOG release but I think there are a few questions still to answer, so I won’t rush into it.

Using CRON and keeping it kind of separte from the other code seems like a good idea to me too. Still wondering if using a UUID is better than clearing the DB and wait for new input once a week. I do like the later approach but would spread the time even more. Across one whole way! Plus we have people from different timezones which adds to the randomization as well. So if we don’t need a UUID, is ther a point in using one?

Making it a installer question I am wondering if many people just use the default No and we lack a fair amount of stats?! Currently we have a global user count polling and inserting information on the FOG web UI logon screen anyhow. Adding general stats like OS version and brand doesn’t make it a real analytics tool (I don’t think we should name it anylytics!) and so I am wondering if there is really a point in asking via the installer as we have done stats since a long time.

[Wayne Workman]

Got a bit more done on this today.

The comparison between my work and the dev-branch can be seen here.

This is what the settings look like:

This is what the cron job looks like - this is dynamically generated when the installer runs:

This is what the logging on your local FOG Server will look like:

I’ll be using the daily installation tests against my own fogproject fork to test against all the major OSs here shortly. I’ll validate the cron jobs get created correctly on each and then I’ll manually change all of the cron jobs to run every minute. This is so I can test immediately rather than waiting a week.

[Sebastian Roth]

@Wayne-Workman Well done! Haven’t had a chance to test but looks fine on first sight.

May I ask you to rename it all. The word “analytics” has just so much background to it - we don’t do real analytics here…

[Wayne Workman]

What do you want it to be called?

Tom had used the “analytics” term in his suggestions which is why I named everything like that, as later on it might be expanded to do more than simply collecting versions. Originally I named everything “popularity”

[Tom Elliott]

@Wayne-Workman, @Sebastian-Roth

I think analytics is the appropriate term, even if the information we’re using is minimal.

I’ve added similar code, slightly revised.

Please take a look and let me know:

https://github.com/FOGProject/fogproject/commit/1435d1b1852e452295359fb045f3bb38f01ba18f
This introduces the information - but I had a typo which is where the “second” link comes in.

AND

https://github.com/FOGProject/fogproject/commit/acfc6445d3b8f9497bf2b63e69e3faec0c51ee51

Thank you,

[Wayne Workman]

@Tom-Elliott Looks fine to me. Should I submit a pull request to the dev-branch since you’ve already added this to the 1.6 branch?

[Tom Elliott]

@Wayne-Workman I think that’d work best, yes.

[Wayne Workman]

I’ve validated the installation code as I’ve written it does work against all the major distributions - and I’ve also discovered my daily test for Ubuntu 20 was really Ubuntu 18 (fixed now).

Below is what the data looks like in my test environment. It’s interesting to note that RedHat changes it’s distribution name from major release to major release. It’s of no consequence for our use, but I think that’s interesting.

The server-side’s code is in this PR:
https://github.com/FOGProject/fog-community-scripts/pull/62

The FOG Server’s code is in this PR:
https://github.com/FOGProject/fogproject/pull/405

[Sebastian Roth]

@Tom-Elliott said in Popularity Contest:

I think analytics is the appropriate term, even if the information we’re using is minimal.

I don’t agree! The term “analytics” implies way more than we do and it might turn down people who use FOG and don’t like analytic stuff that is going on all over the web.

Why should we name it something we don’t want to do - and I am absolutely sure I never add real analytics stuff to FOG myself.

[Wayne Workman]

Maybe we call it “Rainman”?

[Junkhacker]

@Wayne-Workman how about “install survey”

[Tom Elliott]

Analytics:

information resulting from the systematic analysis of data or statistics.

What are we gathering,
OS Name/Type, OS Version, FOG Version

What are we doing?
Analyzing how many of a type of OS and version, along with the number of “versions” run out in the wild.

While the data we’re gathering is, right now, only very basic, I think analytics is still the true term.

If we decide analytics is not the term to use, I still think it’s important to add (and I’m working on doing that) a way for the admins to disable if they don’t want their information being reported. We’re not doing much or collecting any metrics of the data in the sense of IPs or how we can best improve our code base. We are, after all, collecting and sending data to another source.

This information we are gathering is still a form of data we can use to gather what Operating Systems and versions, along with the general FOG version being used in the wild.