Improve documentation

[Sebastian Roth]

@Jurgen-Goedbloed This is really awesome you ask about this!! I was thinking about this the Last days and was planing to ask for help in the forums to improve documentation. So I definitely vote for this! Let’s do it…

Probably best if we create a new repo for the documentation in https://github.com/FOGproject/
I would think.

[Wayne Workman]

@Sebastian-Roth why do you think documentation should be in a separate repo?

[Sebastian Roth]

@Wayne-Workman said in Improve documentation:

why do you think documentation should be in a separate repo?

Good question. Hmm. Just thought it might be easier to keep track of changes when code and docu are not all in one repo.

[Wayne Workman]

Here’s the benefits I can think of for keeping them together.

• a certain version of the documentation is locked together with the corresponding code it’s documenting.
• When you checkout an older version of fog, you’re also checking out the older corresponding documentation, and if you’re checking out a newer fog, you’re checking out newer corresponding documentation.
• When comparing differences between an older version and newer one, it may be easier for some to see the changes by looking at the differences in documentation.

[Sebastian Roth]

@Wayne-Workman Good point. Though I fear we don’t get our act together to really update the documentation accordingly. Having it all in one repo doesn’t force us to actually update it.

On the other hand we can still tag a particular state of the documentation in a separate repo to match that same version in the fogproject code repo.

I just remembered that Joe actually had started a repo and readthedocs project some time ago but we never got into it…

@Jurgen-Goedbloed Do you have an account with github and readthedocs yet?

[A Former User]

I will create these accounts and will let you know.

I have to dig some deeper into sphinx/readdthedocs, but I know many projects have documentation for different versions.

Take Ansible as example: https://docs.ansible.com/ansible/latest/index.html In the top left part you can choose a version. I think this would be handy for Fog as well. A new release: a new set of documentation where you can make the changes that are specific for that version.

[JJ Fullmer]

For the FogApi powershell module I’m using read the docs with mkdocs.
You can see the simple .readthedocs.yml and mkdocs.yml config files in my repo for it here https://github.com/darksidemilk/FogApi
The published version is here https://fogapi.readthedocs.io/en/latest/

Granted, documenting a powershell module is a bit simpler since it’s mostly documenting the functions and what they do and can be generated from comment blocks in-line in the code.

I think readthedocs requires (or is at least by default designed) to have the docs in the same repo to help maintain versions. I don’t have versioning setup though. There’s a page for setting up based on branches and I believe releases and tags as well. I believe it could still be done in a separate repo though, might just require more effort to link versions. Or it could be easier cause you just create a new version each time a version of fog is released and whatever the documentation said at that point is the documentation for that version.

There is also githubpages as a similar option, I found readthedocs a bit easier for my purposes.

I also remember @Joe-Schmitt setting up readthedocs.io for fog-too I think. That’s how I knew what readthedocs was when I wanted to make public documentation for my powershell code.

I would love to help out with getting this setup and some things documented.

Maybe we could have something that grabs stuff from the forums, or maybe just a label/tag we can report against to periodically go through and add to documentation

[A Former User]

I have played with sphinx/github and readthedocs.

I have created an account at github: jgoedbloed. The github account is also used for readthedocs.

If you look at my test project in github: https://github.com/jgoedbloed/docstest
then you’ll see the ‘docs’ folder that contains all documentation (for now this is just a skeleton, no useful information to be found here).

I have set up a readdthedocs account https://jgoedbloed-docstest.readthedocs.io/en/1.1/ and this compiles from the github project and creates the html documentation.

As @JJ-Fullmer states: the nice thing of keeping the docs inside the fogproject source tree is that you can use versions and that each version gets it’s own documentation in readthedocs. As soon as you tag a release in github you get a new version in readthedocs and the documentation for old versions will be left as is,

If you set up a webhook in readthedocs then the documentation automatically gets generated if you commit your work to github.

Agree that this is a good way to proceed?

[Sebastian Roth]

@Jurgen-Goedbloed Sounds good, I just added you as a maintainer for https://readthedocs.org/projects/fogproject/ so you can also edit settings and such.

As mentioned earlier I don’t agree that having documentation within the code repo will make things more clear. It would take a lot of effort and communication to make sure the state of the documentation in the code repo is always up to date (at least when we push for a next release). Ideally this is the case but we are so far from it that I guess this won’t work out.

As a more flexible way we could have the documentation in a separate repo (which is already created) and tag states of the documentation with the exact same names that we use in the code repo (e.g. 1.5.9) when documentation is ready to go. Easy enough to synchronize but not as static as if we are bound to tagging code and docs at the exact same point.

The current plan is to release one final last 1.5.10 version before we move on to 1.6.x version which has a new web UI and some new features that we are working on. I would hope that we can come up with a nice documentation for 1.5.10 some time after the release. Most of that documentation can be re-used for the upcoming 1.6.x stuff as well (beside the screenshots).

@moderators @testers Who of you would like to put in a bit of time to help getting the documentation together?

[Wayne Workman]

At one point @Joe-Schmitt had shared a google doc and several people assisted outlining some documentation there. I can probably dig up the link.

I guess we should know what ought to go into the new documentation. It’s easy to get carried away with all the details concerning Linux and all the open-source components that make up fog. We should control scope for the first effort, keeping the scope small.

I might suggest adding some installation guides - many are already created in the wiki (some even have videos).

A section on DHCP and ProxyDHCP - keeping it lightweight, just what you need to do to get going in the most common scenarios.

I’d suggest a beginners guide to imaging. How to capture an image, how to deploy an image.

Talk about the FOG Client, how to use it for domain joining.

Past that, we can add to it as we need. Small steps.

[A Former User]

@Sebastian-Roth if you think a separate repo is better, than that’s fine for me and it is good to keep it under the fogproject. We can place release tags in that project and achieve the same goal.

@Wayne-Workman yes let’s start small. From my experience with setting up Fog project as a sysadmin/network engineer I am interested in the following parts:

• How does Fog work? A small introduction and a description of components and how it fits in the network. The last part is important for network admins to understand and support a fog installation in their network
• Installation instructions on Debian and Redhat of a fog server and fog storage server
• Managing fog (the Web UI)
• Doing your first image capture/deployment
• Reference of all configuration options

Later we will have to discuss whether to include advanced options like exotic installs, FAQ, Troubleshooting although I think that the forums already play a great role in this and maybe also the wiki.

If you agree, then let me assemble the first pages. The wiki provides lots of information that is very usable, so that’s just copy/paste and reformat.

[Sebastian Roth]

@Jurgen-Goedbloed said in Improve documentation:

If you agree, then let me assemble the first pages. The wiki provides lots of information that is very usable, so that’s just copy/paste and reformat.

Sounds great to me!

From a practical point of view I will organize for you to have write/push access to the fog-docs repo - and everyone else who is keen to work on this (please send me a PM). Sure there is another way by forking that repo to your own account and sending in pull requests but that adds extra work for you and us and I don’t think that’s apropriate.

[JJ Fullmer]

@Sebastian-Roth and @Jurgen-Goedbloed Could I also be added as a maintainer. I would love to help with this project.
Maybe we make some github issues for what documents need creating and or copy pasting and divide and conquer?

[JJ Fullmer]

@Jurgen-Goedbloed
I would actually strongly suggest we use mkdocs instead of sphinx.
mkdocs works better with markdown from what I’ve read and experienced and it’s a lot easier to setup and learn.
Considering the forum uses markdown, the existing wiki uses markdown, I think it would be a better option for the project.
Markdown is also a lot easier to learn and to write and I feel like it would make it a lot easier for future maintainers to pick up and run with.

[Sebastian Roth]

@JJ-Fullmer While Markdown might be easier in the first place there are a few reasons not to use it: https://www.ericholscher.com/blog/2016/mar/15/dont-use-markdown-for-technical-docs/ (linked from readthedocs.io)

[A Former User]

@JJ-Fullmer I have set up the first documentation in sphinx now. I agree with you that it is more difficult than markdown. I have no development experience and needed to learn sphinx from scratch, but having read Sebastian’s comment, let’s start with sphinx. I think when the ‘skeleton’ is set up, it will be much easier to contribute as you have examples.

[JJ Fullmer]

@Sebastian-Roth I just read through that and I see his points but it’s from 4 years ago. From my experience with markdown commonmark is now widely used. Both sphinx and mkdocs use commonmark.
However, considering it sounds like we want the easily navigable versions, sphinx may be the way to go as mkdocs in readthedocs only has one theme that shows the readthedocs version dropdown (I messed with a bunch of themes yesterday). Granted sphinx can be configured to use markdown, but I suppose I can learn another documentation language.

[JJ Fullmer]

@Jurgen-Goedbloed
Are you able to see the built readthedocs page?
I’m having trouble getting it to build locally to see what it’s going to look like and when I go to docs.fogproject.org I get a 403 error. I think that @Joe-Schmitt may have set some security up on that page on the fogproject domain while it was to be under construction. Maybe we should host it on the readthedocs domain until we get that figured out? That’s a setting in the readthedocs project that I can’t access.

edit:
Figured out my build problems, so can at least browse it locally. Updated the readme with build instructions and added a make.ps1 for easier building.

[A Former User]

@JJ-Fullmer No, I’m not. I see the same as you, an nginx 403 page.
I think that for the domain fogproject.org a DNS CNAME record should be made:
docs.fogproject.org IN CNAME readthedocs.io

I have admin access to readthedocs and I have removed this domain temporary until the DNS is fixed. For now you can see the first results under https://fogproject.readthedocs.io

The initial structure is now in github and I think it is best if we spread the documentation across many files, as this decreases the chance of having merge conflicts when together working on the documentation.

[Sebastian Roth]

@Jurgen-Goedbloed said in Improve documentation:

I think that for the domain fogproject.org a DNS CNAME record should be made:
docs.fogproject.org IN CNAME readthedocs.io

We have NGINX running on our webserver. I just had a quick look at the config and for https://docs.fogproject.org we have a proxy_pass (reverse proxy) setting to https://fogproject.readthedocs.io:443

When loading the page I get this in the error logs:

2020/10/07 15:02:08 [error] 26062#26062: *2758 connect() to [2606:4700::6811:2052]:443 failed (101: Network is unreachable) while connecting to upstream, client: 91.18.10.171, server: docs.fogproject.org, request: "GET / HTTP/2.0", upstream: "https://[2606:4700::6811:2052]:443/", host: "docs.fogproject.org"

Interesting it resolves to an IPv6 address but is unable to connect to that. I will take a look at this tomorrow morning.