Improve documentation

[A Former User]

I have played with sphinx/github and readthedocs.

I have created an account at github: jgoedbloed. The github account is also used for readthedocs.

If you look at my test project in github: https://github.com/jgoedbloed/docstest
then you’ll see the ‘docs’ folder that contains all documentation (for now this is just a skeleton, no useful information to be found here).

I have set up a readdthedocs account https://jgoedbloed-docstest.readthedocs.io/en/1.1/ and this compiles from the github project and creates the html documentation.

As @JJ-Fullmer states: the nice thing of keeping the docs inside the fogproject source tree is that you can use versions and that each version gets it’s own documentation in readthedocs. As soon as you tag a release in github you get a new version in readthedocs and the documentation for old versions will be left as is,

If you set up a webhook in readthedocs then the documentation automatically gets generated if you commit your work to github.

Agree that this is a good way to proceed?

[Sebastian Roth]

@Jurgen-Goedbloed Sounds good, I just added you as a maintainer for https://readthedocs.org/projects/fogproject/ so you can also edit settings and such.

As mentioned earlier I don’t agree that having documentation within the code repo will make things more clear. It would take a lot of effort and communication to make sure the state of the documentation in the code repo is always up to date (at least when we push for a next release). Ideally this is the case but we are so far from it that I guess this won’t work out.

As a more flexible way we could have the documentation in a separate repo (which is already created) and tag states of the documentation with the exact same names that we use in the code repo (e.g. 1.5.9) when documentation is ready to go. Easy enough to synchronize but not as static as if we are bound to tagging code and docs at the exact same point.

The current plan is to release one final last 1.5.10 version before we move on to 1.6.x version which has a new web UI and some new features that we are working on. I would hope that we can come up with a nice documentation for 1.5.10 some time after the release. Most of that documentation can be re-used for the upcoming 1.6.x stuff as well (beside the screenshots).

@moderators @testers Who of you would like to put in a bit of time to help getting the documentation together?

[Wayne Workman]

At one point @Joe-Schmitt had shared a google doc and several people assisted outlining some documentation there. I can probably dig up the link.

I guess we should know what ought to go into the new documentation. It’s easy to get carried away with all the details concerning Linux and all the open-source components that make up fog. We should control scope for the first effort, keeping the scope small.

I might suggest adding some installation guides - many are already created in the wiki (some even have videos).

A section on DHCP and ProxyDHCP - keeping it lightweight, just what you need to do to get going in the most common scenarios.

I’d suggest a beginners guide to imaging. How to capture an image, how to deploy an image.

Talk about the FOG Client, how to use it for domain joining.

Past that, we can add to it as we need. Small steps.

[A Former User]

@Sebastian-Roth if you think a separate repo is better, than that’s fine for me and it is good to keep it under the fogproject. We can place release tags in that project and achieve the same goal.

@Wayne-Workman yes let’s start small. From my experience with setting up Fog project as a sysadmin/network engineer I am interested in the following parts:

• How does Fog work? A small introduction and a description of components and how it fits in the network. The last part is important for network admins to understand and support a fog installation in their network
• Installation instructions on Debian and Redhat of a fog server and fog storage server
• Managing fog (the Web UI)
• Doing your first image capture/deployment
• Reference of all configuration options

Later we will have to discuss whether to include advanced options like exotic installs, FAQ, Troubleshooting although I think that the forums already play a great role in this and maybe also the wiki.

If you agree, then let me assemble the first pages. The wiki provides lots of information that is very usable, so that’s just copy/paste and reformat.

[Sebastian Roth]

@Jurgen-Goedbloed said in Improve documentation:

If you agree, then let me assemble the first pages. The wiki provides lots of information that is very usable, so that’s just copy/paste and reformat.

Sounds great to me!

From a practical point of view I will organize for you to have write/push access to the fog-docs repo - and everyone else who is keen to work on this (please send me a PM). Sure there is another way by forking that repo to your own account and sending in pull requests but that adds extra work for you and us and I don’t think that’s apropriate.

[JJ Fullmer]

@Sebastian-Roth and @Jurgen-Goedbloed Could I also be added as a maintainer. I would love to help with this project.
Maybe we make some github issues for what documents need creating and or copy pasting and divide and conquer?

[JJ Fullmer]

@Jurgen-Goedbloed
I would actually strongly suggest we use mkdocs instead of sphinx.
mkdocs works better with markdown from what I’ve read and experienced and it’s a lot easier to setup and learn.
Considering the forum uses markdown, the existing wiki uses markdown, I think it would be a better option for the project.
Markdown is also a lot easier to learn and to write and I feel like it would make it a lot easier for future maintainers to pick up and run with.

[Sebastian Roth]

@JJ-Fullmer While Markdown might be easier in the first place there are a few reasons not to use it: https://www.ericholscher.com/blog/2016/mar/15/dont-use-markdown-for-technical-docs/ (linked from readthedocs.io)

[A Former User]

@JJ-Fullmer I have set up the first documentation in sphinx now. I agree with you that it is more difficult than markdown. I have no development experience and needed to learn sphinx from scratch, but having read Sebastian’s comment, let’s start with sphinx. I think when the ‘skeleton’ is set up, it will be much easier to contribute as you have examples.

[JJ Fullmer]

@Sebastian-Roth I just read through that and I see his points but it’s from 4 years ago. From my experience with markdown commonmark is now widely used. Both sphinx and mkdocs use commonmark.
However, considering it sounds like we want the easily navigable versions, sphinx may be the way to go as mkdocs in readthedocs only has one theme that shows the readthedocs version dropdown (I messed with a bunch of themes yesterday). Granted sphinx can be configured to use markdown, but I suppose I can learn another documentation language.

[JJ Fullmer]

@Jurgen-Goedbloed
Are you able to see the built readthedocs page?
I’m having trouble getting it to build locally to see what it’s going to look like and when I go to docs.fogproject.org I get a 403 error. I think that @Joe-Schmitt may have set some security up on that page on the fogproject domain while it was to be under construction. Maybe we should host it on the readthedocs domain until we get that figured out? That’s a setting in the readthedocs project that I can’t access.

edit:
Figured out my build problems, so can at least browse it locally. Updated the readme with build instructions and added a make.ps1 for easier building.

[A Former User]

@JJ-Fullmer No, I’m not. I see the same as you, an nginx 403 page.
I think that for the domain fogproject.org a DNS CNAME record should be made:
docs.fogproject.org IN CNAME readthedocs.io

I have admin access to readthedocs and I have removed this domain temporary until the DNS is fixed. For now you can see the first results under https://fogproject.readthedocs.io

The initial structure is now in github and I think it is best if we spread the documentation across many files, as this decreases the chance of having merge conflicts when together working on the documentation.

[Sebastian Roth]

@Jurgen-Goedbloed said in Improve documentation:

I think that for the domain fogproject.org a DNS CNAME record should be made:
docs.fogproject.org IN CNAME readthedocs.io

We have NGINX running on our webserver. I just had a quick look at the config and for https://docs.fogproject.org we have a proxy_pass (reverse proxy) setting to https://fogproject.readthedocs.io:443

When loading the page I get this in the error logs:

2020/10/07 15:02:08 [error] 26062#26062: *2758 connect() to [2606:4700::6811:2052]:443 failed (101: Network is unreachable) while connecting to upstream, client: 91.18.10.171, server: docs.fogproject.org, request: "GET / HTTP/2.0", upstream: "https://[2606:4700::6811:2052]:443/", host: "docs.fogproject.org"

Interesting it resolves to an IPv6 address but is unable to connect to that. I will take a look at this tomorrow morning.

[JJ Fullmer]

@Sebastian-Roth @Jurgen-Goedbloed
I believe that fogproject uses cloudflare. There’s a note about using cloudflare dns in the readthedocs docs.
https://docs.readthedocs.io/en/stable/custom_domains.html?highlight=domains#notes-for-cloudflare-dns-users

Doesn’t look like we’re getting that same error as described in their documentation but may still be related.

I think that reverseproxy might be what Joe set up 3 years ago but it looks like that feature is depreciated. https://docs.readthedocs.io/en/stable/custom_domains.html?highlight=domains#proxy-ssl

[Sebastian Roth]

@JJ-Fullmer Thanks for the hint! Seems like reverse proxy setup was used earlier but now readthedocs is able to provide SSL certificates for external domains (interesting concept I find) through cloudflare. We’ll setup a CNAME soon and und the meantime we’ll use https://fogproject.readthedocs.io/

[JJ Fullmer]

@Sebastian-Roth @Wayne-Workman @Jurgen-Goedbloed
So I’m going through the management section of the wiki, trying to do one page/section a day converting them to the new one. Going back through and updating them will be a separate thing I think.
There are some old flash video tutorials hosted in places like http://freeghost.sourceforge.net/videotutorials/hostinfo.html

Do we want to recreate these in a newer not about to be depreciated format? And with the new gui?
We could also just host them in github, especially for ones that don’t require sound and can just be .gif files.

For now I can just embed the old links, but especially these flash ones probably need to be updated or removed, we should just have a standard as we migrate I think. There’s also the factor that the fog gui will change again in 1.6.

[Wayne Workman]

@JJ-Fullmer can they be converted to something YouTube supports? (as opposed to re-recording)

We can put them on the FOG Project Videos YouTube channel. https://www.youtube.com/channel/UCrvOQPcm1SDIfIrzWZ9K3bA/videos

[Sebastian Roth]

@JJ-Fullmer Thanks you are pushing this forward way quicker than I would have imagined.

especially for ones that don’t require sound and can just be .gif files.

While I really like the idea of keeping things as simple as possible I wonder if GIF is helpful with content that you might want to pause or skip through.

There’s also the factor that the fog gui will change again in 1.6.

That’s a good point. We might want to wait with updating the video stuff until 1.6 is on the doorstep. Just my 50 cents.

[JJ Fullmer]

@Wayne-Workman said in Improve documentation:

@JJ-Fullmer can they be converted to something YouTube supports? (as opposed to re-recording)

We can put them on the FOG Project Videos YouTube channel. https://www.youtube.com/channel/UCrvOQPcm1SDIfIrzWZ9K3bA/videos

I think this would be an excellent method, gives us extra search visibility too.

@Sebastian-Roth Good point on the gif. I was just thinking of what I’ve seen in other documentation recently. It’s cool for showing how quick and easy some things are but pause buttons are nice.

Read-the-docs has a youtube video embedded in their docs so I took that as an example and did a test with a random youtube video from our channel
https://fogproject.readthedocs.io/en/latest/management/index.html#hosts

Looks good to me.

I am having trouble getting the swf converted though. When I run it through handbrake or any online convert tool it comes out as a jumbled mess. It’s also a video showing the 0.12 fog gui. Maybe it’s not worth it, I very much doubt anyone uses that version anymore.

[JJ Fullmer]

Well I found one video converter that worked. But it was just the trial of it so I only got half of the video. They want $45 though.
https://www.faasoft.com/video-converter.html

Not sure if that’s worth it for the flash videos of the pre 1.0 fog gui.