Popularity Contest



  • As I sit here working on making the daily fog installation tests more reliable, I wondered to myself when I should remove the older OSs (centos7, rhel7, etc). I thought it’d be nice to know how often what OS is being used to install FOG on. So I propose the following.

    I can create a publicly available API that simply accepts a “string” being submitted. Using this, I can put together a popularity contest type thing that will allow us to see what people are installing FOG on. I can build the API (I have scaffolding that I can use to get this done pretty fast) and run it, and make the results publicly viewable.

    Within the FOG Installer scripts, I can add a question about participating in the popularity contest. I think this should default to “Yes”. If “Yes” is chosen, a simple curl call is executed sending the output of lsb_release --all to the API.

    Thoughts on this? I’m willing to put it all together and submit the pull requests for it. Though wanted feedback before I went after it.


  • Senior Developer

    @george1421 Sorry I have not found enough time to look into this last week. Would have been nice to add this to the upcoming FOG release but I think there are a few questions still to answer, so I won’t rush into it.

    Using CRON and keeping it kind of separte from the other code seems like a good idea to me too. Still wondering if using a UUID is better than clearing the DB and wait for new input once a week. I do like the later approach but would spread the time even more. Across one whole way! Plus we have people from different timezones which adds to the randomization as well. So if we don’t need a UUID, is ther a point in using one?

    Making it a installer question I am wondering if many people just use the default No and we lack a fair amount of stats?! Currently we have a global user count polling and inserting information on the FOG web UI logon screen anyhow. Adding general stats like OS version and brand doesn’t make it a real analytics tool (I don’t think we should name it anylytics!) and so I am wondering if there is really a point in asking via the installer as we have done stats since a long time.



  • @Junkhacker absolutely. In fact, I was planning to make daily dumps of the database available.


  • Developer

    @Wayne-Workman shall we make the stats public, because “why not” and to be transparent about what we collect?



  • I have the information collection portion of the server completed. Code is currently sitting in my fork of fog-community-scripts.

    Here’s what the request looks like right now:

    curl -X POST -H "Content-Type: application/json"  -d '{"fog_version":"12.34.56","os_name":"Debian","os_version":"10"}' http://fog-popularity-contest.theworkmans.us:/api/records
    

    ^ That’s a live link, should work for whoever tries it. If it’s stored ok, the reply is

    {"message":"record recorded"}
    

    Here’s a screenshot of the first test record saved, and the table layout.
    database.png

    I’ll work on the client side next, will follow the conclusion of the guidance provided by everyone here.

    I do plan on adding https to the server side.
    As far as presentation of the data, not sure on that yet. Kicking around ideas of rendering some basic graphs in .png format every day and serving those, as well as making a daily database dump available for download.



  • @Junkhacker I think this is the right move.


  • Developer

    @Wayne-Workman i realize i’m coming into this really late, but why not randomize what day of the week we collect the data as well, and we can just look at the window of “last 7 days” when we look at the data?



  • @developers what branch would you like the PR to be created to?



  • Great approach Tom. If a crontab is setup to only send once per week, the unique identifier becomes totally unnecessary. Because as you suggested, we can delete the collected data on Friday, and have all new fresh data by Monday - or keep all the data, and build reports based on the last week only.

    One issue I see to this approach: having potentially many thousand fog servers all reporting in at the same time I think isn’t going to work out. I’d suggest the minute value of the Crontab be randomized between 0 and 59. This will spread the load out for the receiving API.


  • Senior Developer

    @Sebastian-Roth, @Wayne-Workman, @george1421 I agree.

    I think the GUID generation can be done relatively simply. We may already have a GUID generated at install time. (FOG_UUID in FOG Settings I think.) Though I don’t know how generalized it is, and it’s more a random ID, not an actual UUID (in UUID Standard)

    So my thoughts:

    Keep this “polling/analytics” separate from the main fog code base (Maybe under utils?)

    Ask, on next update/install, if the admin would like to send analytics data to fog, and potentially what to send?

    Store this section to it’s own configuration file.

    So the asking requests something like:

    Would you like to send analytic data for FOG? (y/N)
    We would like to collect the information once a week, on Sunday’s at 03:00:00. (This allows us to delete entries older than a week.)

    The information we would like to collect are:

    • Randomly Generated UUID -> This allows us to track the information anonymously, but also not continually add the request to the queue, data is inserted once, then updated as needed.
    • FOG Version -> This will allow us to track what versions of FOG are currently in use.
    • OS -> This tells us what type of OS you are using to host the fog server.
    • OS Version -> This tells us what version of OS you are using to host the fog server
    • Timestamp -> Just the date and time this was sent.

    We allow the user to select what they’re comfortable sending. I think this is just a better method overall.

    The utility should simply create a simple shell script to send the data the user would like in real time. To collect the FOG Version, should should be able to do a lookup of the /var/www/fog/lib/fog/system.class.php and look up FOG_VERSION in the file.
    OS and OS Version should look at lsb_release -a if the command exists, or look at /etc/os-release, or fallback as needed. Finally, the utility would then create the crontab to perform the task regularly.

    Getting UUID = echo $(cat /proc/sys/kernel/random/uuid)
    We call the configuration file something like /opt/foganalytics/.foganalytics
    The script would be called something like /opt/foganalytics/foganalytics
    Crontab would look like: 0 3 * * 0 /usr/bin/bash /opt/foganalytics/foganalytics >/dev/null 2&>1

    Does this all make sense and sound doable?


  • Senior Developer

    @Wayne-Workman Nice one. Thanks for bringing it up. While I hate all the tracking stuff going on nowadays I think it can be used in a good and totally anonymous way to help improve things.

    FOG already has got this but it might need improvment. I can take a look in the next week or so.

    I am wondering if generating a GUID is really a good idea as it would make tracking possible again. Don’t like the idea. On the other hand we do need some form of ID to distinguish between…



  • George, these are great ideas and questions. This is the beauty of open source. Collaboration.

    I’ve revised my thoughts now. Here they are.

    • Generate a GUID on the FOG system during installation. This will be unique, but not tie to any particular organization.
    • On the fog web login screen, send a request with the GUID, FOG Version, and OS Version details to an API.
    • Request should be non-blocking. If it fails or times out, it should not impact the login process.
    • API records GUID, FOG Version, OS Version, and a datetime stamp.

    These things allow:

    • Not double-counting any system due to the GUID.
    • The ability to know how many are running in the last month/week whatever.
    • How many of each FOG Version is out there currently running
    • How many of each OS is out there running FOG.
    • Presumably see how long a version lives on average, among other things.
    • This protects privacy as the GUID does not tie to any organization, and simply allows us to not double-count entries.

    Thoughts?


  • Moderator

    You know the fog server or the web ui does query (somewhere) to find out the latest version of FOG vs what is installed. Is there a way to send the FOG version number during that query?

    What we really need to know (from a demographics stand point) is

    1. how many active FOG installs are out there running
    2. What version of FOG is currently running
    3. What host OS and version is the #2 running on

    As long as we stick to those values only I don’t see any infringement on anyone’s privacy. There would be no way to tie the organization name to the data. What is really needed is prospective data. I know we could add that to FOG moving forward, but we really need to know what is already out there.

    The other thing is to only support a distro supported OS. When the distro drops support for an OS then FOG should too. i.e. should FOG still support ububntu 12.04 or 14.04? When rhel/centos 7 EOL (hint: June 30th, 2024), does that mean FOG will support centos 7 until 2024? Is that the right thing to do? (asking)


Log in to reply
 

306
Online

7.4k
Users

14.5k
Topics

136.6k
Posts