The future of partclone and therefore FOG as it is

[Sebastian Roth]

@george1421 said in The future of partclone and therefore FOG as it is:

If the two formats are not compatible we might have to supply both version in FOS and then have a key file, db flag, or something to denote the version of partclone being used.

More or less exactly along the line of what I have been thinking about this. But I don’t actually like the idea. It’ll cause us all so much trouble I fear.

[Junkhacker]

the reason the partitions aren’t usable is because of a bug in partclone since at least 3.11 regarding --ignore_crc, which we use in our scripting. if you remove that, it works fine. i’ve been doing testing with partclone 3.12 for a while and it is completely backward compatible with the existing images. sorry i didn’t see this discussion until now.

also, there’s new options in the latest versions of ztst and partclone that i think we need to discuss. interesting and potentially very important options have been opened up.

[Quazz]

@Sebastian-Roth I believe I do have dpkg installed, I think I did that to try and make it work with dpkg instead of having to select rpm or dpkg depending on distro flavor, but dpkg has no knowledge of rpm installed packages and is therefore not usable for that purpose.

Definitely needs to be updated, yes.

I’ll double check the config files for building, but I synced the FOG branch before I started so I’m unsure why that would fail.

Hmmm, looks like my sync may not have worked as I notice some differences, strange. Will have to reset and try again.

edit: New build in progress, will test thursday (tomorrow is holiday)

[george1421]

@Sebastian-Roth Attached are the edits I made to the partclone 0.3.12 package to get it to build in buildroot 19.02.1.

Config.in.txt
partclone.mk.txt

[Wayne Workman]

Didn’t read all the responses, has anyone volunteered to build a tool to convert old images to the new format?

[Junkhacker]

@Wayne-Workman there really isn’t a need to do that

[Wayne Workman]

@Junkhacker just going off of the below quote. Are you suggesting a conversion tool would have no utility?

@Sebastian-Roth said in The future of partclone and therefore FOG as it is:

What that essentially means is that if we also move forward and add Partclone 0.3.x to FOG that would break all existing images I reckon (not tested yet).

[Sebastian Roth]

@Wayne-Workman said in The future of partclone and therefore FOG as it is:

I reckon (not tested yet).

Need to emphasize this! @Junkhacker Has obviously worked on this and tested way more than I have.

[george1421]

@Junkhacker said in The future of partclone and therefore FOG as it is:

@Wayne-Workman there really isn’t a need to do that

Do know what I need to change in the fog back end to allow partclone .0.3.12 to function with FOG and legacy captured images? I already have to create a custom init.xz to include the updated partcone application, I could tweak the bash scripts in this environment if I knew what needed tweaking.

[Junkhacker]

@george1421 if your goal is only to “make it work” all you have to do is remove the --ignore_crc flag

however, there’s a couple things i think we should set as the new defaults for the new version of partclone when capturing

[george1421]

@Junkhacker said in The future of partclone and therefore FOG as it is:

all you have to do is remove the --ignore_crc flag

OK then I have to ask the silliest question possible, why was it there in the first place? One might think you would want crc checks in place.

[george1421]

@Junkhacker said in The future of partclone and therefore FOG as it is:

there’s a couple things i think we should set as the new defaults for the new version of partclone when capturing

Well I’ll then defer to your expertise here. What should we change?

If clonezilla is using 0.3.10+ is there any information we can glean from their project?

[Junkhacker]

@george1421 it’s kinda always been there, but i would assume it’s because we could squeeze a little more performance out of fog by not calculating checksums. also, since the image wouldn’t decompress properly anyway if there was corruption the checksum would only catch anything that went wrong between parclone capturing and it being piped into the compressor. which isn’t likely to have any problems, and if there is there’s a lot more going on than a checksum is going to help with.

[Junkhacker]

@george1421 well, like i alluded to with my last comment, adding checksums doesn’t make sense since we pipe it right into a compressor that will add it’s own checksums.
… actually, you can defer to my post here https://forums.fogproject.org/topic/12750/file-format-and-compression-option-request/ for my arguments for change, but the short of it is:

disable checksums with the flag -a0

the rest of my changes actually have to do with the settings we use for compression, come to think of it.

[Junkhacker]

since you’re updating partclone, you guys plan on updating zstd too, right?

[george1421]

@Junkhacker I have to look into what buildroot is pulling for zstd. But in general what ever buildroot is configured to pull (which is usually the latest) is what gets bundled into FOS.

[Sebastian Roth]

@Junkhacker So far we had version 1.3.3 and going to go to 1.3.5. As George said, just using the versions bundled with Buildroot.

[Junkhacker]

@george1421 awesome. if the changes i want to be made make it in to the final builds, it will allow my university to save terabytes with deduplication.

[george1421]

@Junkhacker I just confirmed that 1.3.5 is what is being built by buildroot.

[Junkhacker]

@george1421 a new feature i want to utilize was added in 1.3.8. 1.4.0 is out now.