Supporting LVM

[Wayne Workman]

This thread is to capture discussion around LVM support and related items.

[Wayne Workman]

My initial thoughts on supporting LVM was bash scripts. Then I started thinking about the tooling that would be involved. Intensive use of many bash parsing techniques that we Linux users know all too well. I thought - geeze why don’t these tools just output JSON for us so we don’t have to do bash-wizardry to get the values?

So here’s where my head-space is now: Fork these partition/LVM information gathering tools and add JSON output options to them. This will be a challenge for me personally, as the tools such as fdisk and lvm are likely written in either C or C++. I want to add something like a --output-as-json argument or similar. This is to eliminate all the complex parsing we currently do. Then, something like JQ could be used to easily pull values from the JSON as needed.

Second, how the disk information for an image is stored. We currently use text files stored with the images in their respective directories on the FOG server. Those files have a structure, yes. But I think JSON would be better.

Considering handling legacy images (non-json) I’ve not thought too much on this yet but my initial thoughts is to convert all the text files during a fog install upgrade. Not getting rid of them, but putting their converted counterparts in with the images; leaving the old files in-place. And the upgraded fog system just wouldn’t use the old files.

Looking for some thoughts on this approach before I dive in deep.

[Sebastian Roth]

@Wayne-Workman Sorry I have not answered this yet. It’s been very busy over the last weeks and I hardly get to FOG even now.

So here’s where my head-space is now: Fork these partition/LVM information gathering tools and add JSON output options to them. This will be a challenge for me personally, as the tools such as fdisk and lvm are likely written in either C or C++. I want to add something like a --output-as-json argument or similar. This is to eliminate all the complex parsing we currently do. Then, something like JQ could be used to easily pull values from the JSON as needed.

While I like the idea in general I think it’s a hell of a job to do for several different tools. I could be wrong but that is what my guts tell me. This is me digging through C/C++ code and also a tiny bit of coding on and off for at lease 15 years.

Considering handling legacy images (non-json) I’ve not thought too much on this yet but my initial thoughts is to convert all the text files during a fog install upgrade. Not getting rid of them, but putting their converted counterparts in with the images; leaving the old files in-place. And the upgraded fog system just wouldn’t use the old files.

Sounds like a good strategy to convert files and keep the original ones as well.

All in all this is a huge project you are aiming for. Before you get into it we might think about different options we have and then compare pros and cons.

[Wayne Workman]

@sebastian-roth I’ve been thinking on your comments since you’ve posted them.

I have another idea that would achieve the same ends. A python wrapper. I really, really want to avoid the complexities involved with BASH based parsing. I’m just completely turned off by it, and not interested. Really, I believe firmly all CLI tools ought to have a JSON output option. Seriously! (Maybe I should start a movement?)

Though, the tools do not currently output JSON. I’m skilled with Python and could write a wrapper for these tools, to convert the output to a nice JSON format. Thinking we can call a python script that collects all necessary partitioning information, and spits it all out via stdout in JSON format.

The one hitch - can we get python3 into FOS? Maybe it’s already there? Haven’t checked.

I don’t want to abandon this effort. I’m just looking for a way forward that is both attainable as well as not miserable.

[Sebastian Roth]

@wayne-workman While a wrapper is adding complexity and therefore failures I still think we should look into this. Sounds strange but I am prone to BASH parsing violence. Though I totally understand this is not sexy and causes issues too.

Really, I believe firmly all CLI tools ought to have a JSON output option. Seriously! (Maybe I should start a movement?)

That would be really cool! Hope my last message didn’t turn you down. You are more than welcome to just pick one tool and give it a try. Start coding and find out if you can get this pushed into the upstream code as well. It’s always a win for the OSS community and not a waste of time! After this first one there is a good chance we/you know if it’s possible to do for all the tools needed.

The one hitch - can we get python3 into FOS? Maybe it’s already there? Haven’t checked.

Don’t think we have it included yet (config reference. But that’s no big issue. We can add that and build new inits. Just let me know if you want to go this route or try the other one first?

On the downside Python(3) will surely add to the size of the inits and slow down the PXE boot process a bit. Not sure how much yet. Though we’ll do it if that’s the cost for adding LVM support!

[Wayne Workman]

Another thing I just now thought of - versioning the partition file we keep with the images. We can start out with v1.0.0. And if we change how we store that file, we simply increment the version. This would give us some more flexibility in regard to changes.

Thinking further, the scripts that collect the physical partitions are fine, we can just add JSON output for those.

And really, the LVM describing tools are what needs JSON output, so the effort there is targeted to just those.

I think I’m going to take a swing at adding JSON to an LVM describing tool (there are several).