The future of partclone and therefore FOG as it is

[Sebastian Roth]

Sorry if this sounds a bit daunting. Not meant to. I just stumbled upon a changelog message that I would like to discuss with all the FOG users out there:

From https://clonezilla.org/downloads/stable/changelog.php (Clonezilla and Partclone are kind of partner projects and as I simply couldn’t find release notes on Partclone itself I need to reference this one!):

Clonezilla live 2.5.3-3
…
Partclone was updated to 0.3.8. //NOTE// New image format is used in this release. It is different from the one saved by Partclone 0.2.x.

Within FOG we currently use version 0.2.89 which seems to be the latest 0.2.x release. That’s fine from my point of view even though it’s a bit dated. Updating our whole FOS base system to the latest Buildroot version I ran into a compiling error with Partclone as it still uses ustat function which is deprecated in up-to-date glibc version (2.28). So we seem to get to a point where we need to patch Partclone 0.2.89 to be able to further use it within FOG or we need to come up with a way on how to move to version 0.3.x…

One would think that newer versions of Partclone are able to read the old image format but there are hints on the web that it can cause trouble: https://sourceforge.net/p/clonezilla/discussion/Clonezilla_live/thread/34023334/

Keep in mind that Partclone versions 0.3.x still are proposed as unstable on the official website. On the other hand it’s in use with current Clonezilla releases since almost two years!

If you can’t follow all the tech details above, don’t worry about it too much. The basic message is Partclone seems to move on using a different image format not compatible with what we used to have! What that essentially means is that if we also move forward and add Partclone 0.3.x to FOG that would break all existing images I reckon (not tested yet).

@Quazz Would you find the time to compile FOS builds with Partclone 0.3.x for testing?

[george1421]

I have my dev environment still setup so I can build it if needed. I just need to know what version of buildroot you are building against and I’ll need the current config file for buildroot.

[Quazz]

@george1421 Best thing is probably to just pull the latest FOS from git and build that with an updated partclone.

@Sebastian-Roth I’ll see if I have some time this week.

[Sebastian Roth]

@george1421 Cool thanks. Version os 2019.02.1 and current config is here 64 bit and 32 bit.

But we also need to modify partclone.mk… will try to give you more details on this later.

[Quazz]

@Sebastian-Roth Btw I think that it’s not so much a “new image format”, but rather the checksum mechanism as per Junkhacker https://github.com/Thomas-Tsai/partclone/issues/121 (and https://forums.fogproject.org/topic/12750/file-format-and-compression-option-request/7) ,but I could be wrong of course.

I’ll be testing 3.12 (in the testing branch)

edit: Implemented Junkhacker’s suggestions. (well, only the checksum bits, since we want it to work before doing extra stuff)

Currently building, will test tomorrow.

edit 2: Seems like build errors complaining about legacy options, will see if I can fix that before I go home.

[Sebastian Roth]

@Quazz said:

Seems like build errors complaining about legacy options, will see if I can fix that before I go home.

Are you surely using the new configs I pushed to the repo? 64 bit and 32 bit - those shouldn’t cause any legacy option issues.

But if they do, simply manually run:

cd fssource...
make ARCH... menuconfig

Go to the last menu item (legacy…) and deselect every option that might still be selected. Quit, save, make ARCH...

[george1421]

@Sebastian-Roth I’m currently running a debug deploy with the new inits and partclone 0.3.12. I had to fix the path in partclone.mk as well as remove the 0.2.98 patch. For my centos build environment I also needed to setup epel to install dpkg which is used by the partclone make program to determine the architecture of the system. I don’t know if this is going to be a problem for the 32 build or not.

In the partclone Config.in file I had to comment out the prerequsites so that partclone would build. I don’t know if these settings have been deprecated or not.

 # depends on !(BR2_LARGEFILE && BR2_USE_WCHAR)

I could not find a current reference to BR2_LARGEFILE . BR2_USE_WCHAR was already set to yes so I didn’t worry about that when I commended out that dependency.
I have to be honest I merged the buildroot and fos repositories to build out a buildroot image like I’m accustom. So the fos build.sh may address all of these things. (??). But from initial testing it appears partclone can read a uefi image created with 0.2.98. I’ll know when its done imaging if it boots or not.

[george1421]

@george1421 Well a follow up on this shows that while the partitoins are on the disk, they are not in any usable state. I can’t even mount the partitions. It’s possible that I did something to mess up partclone or the formats are incompatible. I’ll try some different things to see if I can tell which is which.

[Sebastian Roth]

@george1421 Thanks heaps for testing. I think you are on the right track and doing a great job!

Just a few comments: sure 0.2.89 patch wouldn’t work anymore, good to know that we need dpkg and therefore epel to build on CentOS (might need to address this in build.sh - @Quazz yours is working on RHEL without epel?), BR2_LARGEFILE was removed long time ago (we should have fixed that in Config.in as well), yes build.sh kind of merges the files into the buildroot source before compiling.

Well a follow up on this shows that while the partitoins are on the disk, they are not in any usable state. I can’t even mount the partitions.

That sounds horrible!

[george1421]

@Sebastian-Roth Tomorrow I will build a new reference image and capture it with the updated partclone to see if I can capture and deploy from the same version.

The other thing I was thinking (more like random thoughts), if on the fog server there was both the old and new versions of partclone could we create a conversion script to pipe the output of the old partclone to the input of the new partclone (decompressing and recompressing on each end). I know that would put a load on the fog server that it wouldn’t normally see. If the two formats are not compatible we might have to supply both version in FOS and then have a key file, db flag, or something to denote the version of partclone being used.

[Sebastian Roth]

@george1421 said in The future of partclone and therefore FOG as it is:

If the two formats are not compatible we might have to supply both version in FOS and then have a key file, db flag, or something to denote the version of partclone being used.

More or less exactly along the line of what I have been thinking about this. But I don’t actually like the idea. It’ll cause us all so much trouble I fear.

[Junkhacker]

the reason the partitions aren’t usable is because of a bug in partclone since at least 3.11 regarding --ignore_crc, which we use in our scripting. if you remove that, it works fine. i’ve been doing testing with partclone 3.12 for a while and it is completely backward compatible with the existing images. sorry i didn’t see this discussion until now.

also, there’s new options in the latest versions of ztst and partclone that i think we need to discuss. interesting and potentially very important options have been opened up.

[Quazz]

@Sebastian-Roth I believe I do have dpkg installed, I think I did that to try and make it work with dpkg instead of having to select rpm or dpkg depending on distro flavor, but dpkg has no knowledge of rpm installed packages and is therefore not usable for that purpose.

Definitely needs to be updated, yes.

I’ll double check the config files for building, but I synced the FOG branch before I started so I’m unsure why that would fail.

Hmmm, looks like my sync may not have worked as I notice some differences, strange. Will have to reset and try again.

edit: New build in progress, will test thursday (tomorrow is holiday)

[george1421]

@Sebastian-Roth Attached are the edits I made to the partclone 0.3.12 package to get it to build in buildroot 19.02.1.

Config.in.txt
partclone.mk.txt

[Wayne Workman]

Didn’t read all the responses, has anyone volunteered to build a tool to convert old images to the new format?

[Junkhacker]

@Wayne-Workman there really isn’t a need to do that

[Wayne Workman]

@Junkhacker just going off of the below quote. Are you suggesting a conversion tool would have no utility?

@Sebastian-Roth said in The future of partclone and therefore FOG as it is:

What that essentially means is that if we also move forward and add Partclone 0.3.x to FOG that would break all existing images I reckon (not tested yet).

[Sebastian Roth]

@Wayne-Workman said in The future of partclone and therefore FOG as it is:

I reckon (not tested yet).

Need to emphasize this! @Junkhacker Has obviously worked on this and tested way more than I have.

[george1421]

@Junkhacker said in The future of partclone and therefore FOG as it is:

@Wayne-Workman there really isn’t a need to do that

Do know what I need to change in the fog back end to allow partclone .0.3.12 to function with FOG and legacy captured images? I already have to create a custom init.xz to include the updated partcone application, I could tweak the bash scripts in this environment if I knew what needed tweaking.

[Junkhacker]

@george1421 if your goal is only to “make it work” all you have to do is remove the --ignore_crc flag

however, there’s a couple things i think we should set as the new defaults for the new version of partclone when capturing