1.5.7.89: partclone doesn't capture an image in dd mode: wrong options in fog.upload

[Tom Elliott]

@george1421 the -a0 is partclone 0.3.12 argument to disable crc checksumming IIRC. However, the inits downloaded by default are still using 0.2.89 partclone which did not have this argument available. Simply replacing the funcs.sh into these inits will cause issues.

As for LVM imaging, for all @developers, the inits already have functions created that should enable imaging of LVM volumes. The functions exist but I’ve not tested them. I believe they’re in the bottom of funcs.sh and the actual imaging scripts would need to be modified to look for and capture/deploy these volumes. They’ve never been tested, so may need to be and be reworked as needed. The lvm functions have been in there since around 1.4.4 was released.

[Tom Elliott]

@george1421 et al,

https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh#L2298

This is where the LVM stuff resides

[Tom Elliott]

and when it was actually added, January 2017 yay

https://github.com/FOGProject/fogproject/commit/224147284ad83f525e2c198fb5fbaaf1484bc171

[george1421]

In researching this issue I’m finding that this bit of code needs some attention in general.
ref: https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload#L196

        dd)
            imgpart="$imagePath/$img"
            mkfifo /tmp/pigz1 >/dev/null 2>&1
            uploadFormat "/tmp/pigz1" "$imgpart"
            partclone.imager -c -s "$hd" -O /tmp/pigz1 -N -f 1 -a0
            rm /tmp/pigz1 >/dev/null 2>&1
            clearScreen
            ;;

Where I was getting tripped up was thinking that -a0 was related to the -aX0 command from junkhacker’s request here: https://forums.fogproject.org/topic/13206/the-future-of-partclone-and-therefore-fog-as-it-is/98

It took me a bit to figure out what this code is doing, its pretty ingenious on how its using the fifo file and still compressing the image before uploading.

I can’t test the current build for partclone where I’m at so I can’t confirm if the -a0 here is even valid. I’ll be in a better position tomorrow to test the current inits.

[shruggy]

I’m not sure, but I believe that partclone.imager does have the -c option. It’s probably -a0 that got it tripped. (I haven’t checked it though). But partclone.dd has neither -c nor -a. Despite partclone.dd -h happily listing the last option. This is from the current partclone.c source:

if CHKIMG
	static const char *sopt = "-hvd::L:s:f:CFiBz:Nn:";
#elif RESTORE
	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
#elif DD
	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
#else
	static const char *sopt = "-hvd::L:cx:brDo:O:s:f:RCFINiqWBz:E:a:k:Kn:Tt";
#endif

[shruggy]

OK, partclone.imager seems to not have the -c option either. My trick works when capturing just one partition (or the whole disk) in dd mode, but to capture only the LVM volume in dd mode while allowing other modes for the rest of the disk, we’ll probably need something like:

diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
index 5d6054f..050ace9 100644
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
@@ -2049,6 +2049,13 @@ savePartition() {
     getPartType "$part"
     local ebrfilename=""
     local swapuuidfilename=""
+    local clone="-c"
+    local checksum_mode="-a0"
+    if [ "0x8e" == "$parttype" ]; then
+        fstype=imager
+       clone=
+       checksum_mode=
+    fi
     case $fstype in
         swap)
             echo " * Saving swap partition UUID"
@@ -2068,7 +2075,7 @@ savePartition() {
                     debugPause
                     imgpart="$imagePath/d${disk_number}p${part_number}.img"
                     uploadFormat "$fifoname" "$imgpart"
-                    partclone.$fstype -n "Storage Location $storage, Image name $img" -cs $part -O $fifoname -Nf 1 -a0
+                    partclone.$fstype -n "Storage Location $storage, Image name $img" $clone -s $part -O $fifoname -N -f1 $checksum_mode
                     exitcode=$?
                     case $exitcode in
                         0)

Will test it tomorrow.

[Quazz]

To clarify -a0 disables the checksum; junkhacker in the original thread made a small mistake with -aX0 as the X was a placeholder that needed to be replaced with either a 0 or 1 for it to work. It shouldn’t cause any issues and removing it will break compatibility with images made under older versions of partclone.

partclone.dd not having a in its short options is actually a bug in my opinion as per

https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L352 clearly listing several options that are for some reason not included in the sopt list for dd.

That said; for dd specifically it turns out we can ignore that potential bug and remove a0 as per https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L402 setting the checksum_mode to 0 anyway.

And you are right that c is not an option for dd (anymore); I’m guesssing it was removed since a raw copy is always a clone so it makes no sense to specify it.

All that said, I think proper handling of LVM as per what Tom said should be the way forward.

[Sebastian Roth]

@Tom-Elliott Thanks heaps for pointing out that LVM support was prepared for some time ago! Didn’t know that.

However, the inits downloaded by default are still using 0.2.89 partclone which did not have this argument available.

That’s only true for 1.5.7 but people installing dev-branch (as posted here) already use inits with partclone version 0.3.12 (see here).

@shruggy Though I am not exactly sure I would imagine that switching from partclone 0.2.89 to 0.3.12 broke LVM raw capture.

@Quazz said:

All that said, I think proper handling of LVM as per what Tom said should be the way forward.

I’d think so too! The code being prepared for LVM already will make things easier I suppose and we might even add this “feature” before the next release as it turns out that newer versions break LVM (raw!) capture.

[Sebastian Roth]

@Quazz @shruggy Ok, after a major debugging session I am fairly sure I found why partclone.imager doesn’t work anymore the way we are used to it. Find new details here: https://github.com/FOGProject/fos/issues/35

On the question why we use partclone.imager instead of partclone.dd - also discussed in the other forum topic but just to give you the full picture: Intially partclone only had the dd mode that would save sector-per-sector (dd style) images that you’d need to restore using partclone.dd as well. But as partclone.imager was added it made things for us a bit easier to simply use partclone.restore for all images.

[shruggy]

I’ve built a new (or rather old) init.xz. It’s here. I didn’t have time and wanted to get things done quickly, so as a stop-gap measure I just replaced partclone 0.3.12 with the old version 0.2.89 0.2.91. Besides, the image also has some other differences (that hopefully will speed up testing it):

1. Buildroot 2019.11
2. Processor architecture set to Westmere
3. Partclone at 0.2.89 0.2.91
4. Zstd 1.4.4 (it’s really fast just as release notes promised)
5. MDadm 4.1
6. GPTfdisk 1.0.4

The whole diff is somewhat misleading, but nevertheless can be found here.

With it, I captured the disk from my master (still using FOG 1.5.7.89) and successfully restored it to another machine.

As @Sebastian-Roth has already found the root of the problem it’s probably unneeded anymore, but after this I was planning to update partclone on the image to the latest git snapshot on the release branch (that reports itself as 0.2.91). Done.

[Sebastian Roth]

@shruggy Thanks for sharing your work here! You might even think about joining the team and working on the official builds if you are keen.

Just a few comments:

Buildroot 2019.11

We usually stick to the LTS versions for the official builds. Just FYI.

Processor architecture set to Westmere

Do you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

Partclone at 0.2.89

You might want to use the proper patch file for that version for the web UI status bar to work properly.

Zstd 1.4.4 (it’s really fast just as release notes promised)
MDadm 4.1
GPTfdisk 1.0.4

Nice, will take a look at those and update our official builds as well.

BR2_KERNEL_HEADERS_4_19=y

Using this will make your init fail to run with older versions of the Linux kernel. Not a problem for you I suppose but we try to keep compatibility as good as possible.

As @Sebastian-Roth has already found the root of the problem it’s probably unneeded anymore, but after this I planned to update partclone on the image to the latest git snapshot on the release branch (that reports itself as 0.2.91).

I didn’t know the release branch is still on going… I thought it would be 0.2.89 -> 0.3.x

[shruggy]

@Sebastian-Roth

Processor architecture set to Westmere

Do you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

It definitely wouldn’t: Westmere are Intel Core processors from year 2010 IIRC.

Partclone at 0.2.89

You might want to use the proper patch file for that version for the web UI status bar to work properly.

Yes, i found and applied it, just failed to mention (or rather did it very clumsily: that bit about misleading diff).

[george1421]

@Sebastian-Roth said in 1.5.7.89: partclone doesn't capture an image in dd mode: wrong options in fog.upload:

o you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

FWIW: I had to build a one off kernel to image a 486 for a fog admin. So even with the stock bzImage32 anything less than a 586 isn’t supported.

We really don’t have any data on if there is a performance boost from making the minimum processor requirements to a specific model. It would be an interesting test. But the kernel and the inits would need to be in sync.

My gut reaction would be that it would cause more help desk calls about incompatible processors than any performance boost we might get from using newer instruction calls. But as they say, without data its only an opinion.

[shruggy]

Uploaded new version of the FOS image in my post: replaced partclone 0.2.89 with partclone 0.2.91 (aka the latest commit in release branch).

• checks for image version (currently 0001) and displays it on progress screen
• updates translations
• fixes compile issue on CentOS 6.8
• fixes --disable-* options behavior for configure
• updates Makefile
• additional checks for malloc as well as different sizes recorded in an image (used blocks, total blocks, device size)

[shruggy]

I tested and uploaded another version: this time it’s the latest Partclone with the fix suggested by @Sebastian-Roth, i.e.

diff --git a/src/ddclone.c b/src/ddclone.c
index 04f8775..cab5386 100644
--- a/src/ddclone.c
+++ b/src/ddclone.c
@@ -30,7 +30,9 @@ void read_super_blocks(char* device, file_system_info* fs_info)
        }
        strncpy(fs_info->fs, raw_MAGIC, FS_MAGIC_SIZE);
        fs_info->block_size  = PART_SECTOR_SIZE;
-       //fs_info->device_size = get_partition_size(&src);
+#ifdef IMG
+       fs_info->device_size = get_partition_size(&src);
+#endif
        fs_info->totalblock  = fs_info->device_size / PART_SECTOR_SIZE;
        fs_info->usedblocks  = fs_info->device_size / PART_SECTOR_SIZE;
        close(src);

The symbol IMG is defined in Makefile when building partclone.imager and prevents this change from being built into partclone.dd. For now, I’ve successfully captured a disk from one machine and restored it to another one. After the coming Microsoft Patch Day (probably over the next weekend) I am planning to capture another disk image with this and deploy it to my pool in multi-cast mode.

This time, I also built the 64-bit kernel 4.19.94 and used it together with this image. It’s here.

[Sebastian Roth]

Marking as fixed as we have this added to dev-branch and it will be in the next release.