The future of partclone and therefore FOG as it is

[Quazz]

@Sebastian-Roth Btw I think that it’s not so much a “new image format”, but rather the checksum mechanism as per Junkhacker https://github.com/Thomas-Tsai/partclone/issues/121 (and https://forums.fogproject.org/topic/12750/file-format-and-compression-option-request/7) ,but I could be wrong of course.

I’ll be testing 3.12 (in the testing branch)

edit: Implemented Junkhacker’s suggestions. (well, only the checksum bits, since we want it to work before doing extra stuff)

Currently building, will test tomorrow.

edit 2: Seems like build errors complaining about legacy options, will see if I can fix that before I go home.

[Sebastian Roth]

@Quazz said:

Seems like build errors complaining about legacy options, will see if I can fix that before I go home.

Are you surely using the new configs I pushed to the repo? 64 bit and 32 bit - those shouldn’t cause any legacy option issues.

But if they do, simply manually run:

cd fssource...
make ARCH... menuconfig

Go to the last menu item (legacy…) and deselect every option that might still be selected. Quit, save, make ARCH...

[george1421]

@Sebastian-Roth I’m currently running a debug deploy with the new inits and partclone 0.3.12. I had to fix the path in partclone.mk as well as remove the 0.2.98 patch. For my centos build environment I also needed to setup epel to install dpkg which is used by the partclone make program to determine the architecture of the system. I don’t know if this is going to be a problem for the 32 build or not.

In the partclone Config.in file I had to comment out the prerequsites so that partclone would build. I don’t know if these settings have been deprecated or not.

 # depends on !(BR2_LARGEFILE && BR2_USE_WCHAR)

I could not find a current reference to BR2_LARGEFILE . BR2_USE_WCHAR was already set to yes so I didn’t worry about that when I commended out that dependency.
I have to be honest I merged the buildroot and fos repositories to build out a buildroot image like I’m accustom. So the fos build.sh may address all of these things. (??). But from initial testing it appears partclone can read a uefi image created with 0.2.98. I’ll know when its done imaging if it boots or not.

[george1421]

@george1421 Well a follow up on this shows that while the partitoins are on the disk, they are not in any usable state. I can’t even mount the partitions. It’s possible that I did something to mess up partclone or the formats are incompatible. I’ll try some different things to see if I can tell which is which.

[Sebastian Roth]

@george1421 Thanks heaps for testing. I think you are on the right track and doing a great job!

Just a few comments: sure 0.2.89 patch wouldn’t work anymore, good to know that we need dpkg and therefore epel to build on CentOS (might need to address this in build.sh - @Quazz yours is working on RHEL without epel?), BR2_LARGEFILE was removed long time ago (we should have fixed that in Config.in as well), yes build.sh kind of merges the files into the buildroot source before compiling.

Well a follow up on this shows that while the partitoins are on the disk, they are not in any usable state. I can’t even mount the partitions.

That sounds horrible!

[george1421]

@Sebastian-Roth Tomorrow I will build a new reference image and capture it with the updated partclone to see if I can capture and deploy from the same version.

The other thing I was thinking (more like random thoughts), if on the fog server there was both the old and new versions of partclone could we create a conversion script to pipe the output of the old partclone to the input of the new partclone (decompressing and recompressing on each end). I know that would put a load on the fog server that it wouldn’t normally see. If the two formats are not compatible we might have to supply both version in FOS and then have a key file, db flag, or something to denote the version of partclone being used.

[Sebastian Roth]

@george1421 said in The future of partclone and therefore FOG as it is:

If the two formats are not compatible we might have to supply both version in FOS and then have a key file, db flag, or something to denote the version of partclone being used.

More or less exactly along the line of what I have been thinking about this. But I don’t actually like the idea. It’ll cause us all so much trouble I fear.

[Junkhacker]

the reason the partitions aren’t usable is because of a bug in partclone since at least 3.11 regarding --ignore_crc, which we use in our scripting. if you remove that, it works fine. i’ve been doing testing with partclone 3.12 for a while and it is completely backward compatible with the existing images. sorry i didn’t see this discussion until now.

also, there’s new options in the latest versions of ztst and partclone that i think we need to discuss. interesting and potentially very important options have been opened up.

[Quazz]

@Sebastian-Roth I believe I do have dpkg installed, I think I did that to try and make it work with dpkg instead of having to select rpm or dpkg depending on distro flavor, but dpkg has no knowledge of rpm installed packages and is therefore not usable for that purpose.

Definitely needs to be updated, yes.

I’ll double check the config files for building, but I synced the FOG branch before I started so I’m unsure why that would fail.

Hmmm, looks like my sync may not have worked as I notice some differences, strange. Will have to reset and try again.

edit: New build in progress, will test thursday (tomorrow is holiday)

[george1421]

@Sebastian-Roth Attached are the edits I made to the partclone 0.3.12 package to get it to build in buildroot 19.02.1.

Config.in.txt
partclone.mk.txt

[Wayne Workman]

Didn’t read all the responses, has anyone volunteered to build a tool to convert old images to the new format?

[Junkhacker]

@Wayne-Workman there really isn’t a need to do that

[Wayne Workman]

@Junkhacker just going off of the below quote. Are you suggesting a conversion tool would have no utility?

@Sebastian-Roth said in The future of partclone and therefore FOG as it is:

What that essentially means is that if we also move forward and add Partclone 0.3.x to FOG that would break all existing images I reckon (not tested yet).

[Sebastian Roth]

@Wayne-Workman said in The future of partclone and therefore FOG as it is:

I reckon (not tested yet).

Need to emphasize this! @Junkhacker Has obviously worked on this and tested way more than I have.

[george1421]

@Junkhacker said in The future of partclone and therefore FOG as it is:

@Wayne-Workman there really isn’t a need to do that

Do know what I need to change in the fog back end to allow partclone .0.3.12 to function with FOG and legacy captured images? I already have to create a custom init.xz to include the updated partcone application, I could tweak the bash scripts in this environment if I knew what needed tweaking.

[Junkhacker]

@george1421 if your goal is only to “make it work” all you have to do is remove the --ignore_crc flag

however, there’s a couple things i think we should set as the new defaults for the new version of partclone when capturing

[george1421]

@Junkhacker said in The future of partclone and therefore FOG as it is:

all you have to do is remove the --ignore_crc flag

OK then I have to ask the silliest question possible, why was it there in the first place? One might think you would want crc checks in place.

[george1421]

@Junkhacker said in The future of partclone and therefore FOG as it is:

there’s a couple things i think we should set as the new defaults for the new version of partclone when capturing

Well I’ll then defer to your expertise here. What should we change?

If clonezilla is using 0.3.10+ is there any information we can glean from their project?

[Junkhacker]

@george1421 it’s kinda always been there, but i would assume it’s because we could squeeze a little more performance out of fog by not calculating checksums. also, since the image wouldn’t decompress properly anyway if there was corruption the checksum would only catch anything that went wrong between parclone capturing and it being piped into the compressor. which isn’t likely to have any problems, and if there is there’s a lot more going on than a checksum is going to help with.

[Junkhacker]

@george1421 well, like i alluded to with my last comment, adding checksums doesn’t make sense since we pipe it right into a compressor that will add it’s own checksums.
… actually, you can defer to my post here https://forums.fogproject.org/topic/12750/file-format-and-compression-option-request/ for my arguments for change, but the short of it is:

disable checksums with the flag -a0

the rest of my changes actually have to do with the settings we use for compression, come to think of it.