1.5.7.89: partclone doesn't capture an image in dd mode: wrong options in fog.upload

[shruggy]

After upgrade to the latest dev-branch, partclone stopped to capture images in dd mode. My setup:
dual-boot Windows 10 / CentOS 6 (LVM).
lsblk -f output:

NAME   FSTYPE      LABEL             UUID
sda                                                                                     
|-sda1 ntfs        System-reserviert 6A1494EA1494BB11                                   
|-sda2 ntfs                          4C5E04DB5E04BFA6                                   
|-sda3 ext2                          4aef9246-b9cb-41b2-889f-2cd08fc5e966               
`-sda4 LVM2_member                   gYEf27-xHDD-5oCl-qjHi-Uyvy-ScAv-BPAS86

lsblk output:

NAME   MAJ:MIN RM   SIZE RO TYPE
sda      8:0    0   477G  0 disk
|-sda1   8:1    0   500M  0 part
|-sda2   8:2    0   238G  0 part
|-sda3   8:3    0 105.9M  0 part
`-sda4   8:4    0   238G  0 part

The NTFS partitions get captured normally, so does the Ext2 partition, but when it comes to LVM (/dev/sda4), the partclone ends immediately without any error with the resulting size of partition image being almost zero. Almost because of compression overhead (70 bytes or so), but partclone.info reports the size of the image as zero.
/var/log/partclone.log (after doing debug capture):

Partclone v0.3.13 http://partclone.org
Starting to clone device (/dev/sda) to image (/tmp/pigz1)
UID is root.
source=/dev/sda, target=/tmp/pigz1
open source file/device /dev/sda
open target file/device /tmp/pigz1
Initiate image options - version 0002
Initial image hdr - get Super Block from partition
Reading Super Block
open source file/device /dev/sda
0 blocks per checksum
memory needed: 2097152 bytes
bitmap 0 bytes, blocks 2*1048576 bytes, checksum 0 bytes
initial main bitmap pointer 0x522d20
Initial image hdr - read bitmap table
Calculating bitmap... Please wait...
check main bitmap pointer 0x522d20
Writing super block and bitmap...
io_all: write 110, 0 left.
io_all: write 4, 0 left.
done!
print image information
MODE: clone
DEBUG: 3
SOURCE: /dev/sda
TARGET: /tmp/pigz1
OVERWRITE: 1
RESCUE: 0
CHECK: 1
QUIET: 0
FRESH: 1
FORCE: 0
BTFILES: 0
NCURSES: 0
OFFSET DOMAIN: 0x0
CHECKSUM: NONE
CS SIZE: 0
BLOCKS/CS: 0
NOTE:
File system:  raw
Device size:    0 Byte = 0 Blocks
Space in use:   0 Byte = 0 Blocks
Free Space:     0 Byte = 0 Blocks
Block size:   512 Byte
Initial Progress bar
#
Buffer capacity = 2048, Blocks per cs = 0
#
Total block 0
start backup data...
Syncing... OK!
Partclone successfully cloned the device (/dev/sda) to the image (/tmp/pigz1)

What I tried:

1. rebuilt FOS with the latest partclone (that’s why the log reports partclone 0.3.13):

diff --git a/Buildroot/package/partclone/partclone.mk b/Buildroot/package/partclone/partclone.mk
index 07c2022..3035cf2 100644
--- a/Buildroot/package/partclone/partclone.mk
+++ b/Buildroot/package/partclone/partclone.mk
@@ -4,9 +4,9 @@
 #
 ################################################################################

-PARTCLONE_VERSION = 0.3.12
+PARTCLONE_VERSION = 58d138da6e0f473acd80a7c9b8544d104377d50f
 PARTCLONE_SOURCE = partclone-$(PARTCLONE_VERSION).tar.gz
-PARTCLONE_SITE = http://partclone.nchc.org.tw/download/testing
+PARTCLONE_SITE = $(call github,Thomas-Tsai,partclone,$(PARTCLONE_VERSION))
 PARTCLONE_INSTALL_STAGING = YES
 PARTCLONE_AUTORECONF = YES
 PARTCLONE_DEPENDENCIES += attr e2fsprogs libgcrypt lzo xz zlib xfsprogs ncurses host-pkgconf

2. changed partclone.imager to partclone.dd in /bin/fog.upload
   Here I got the message from partclone.dd that it doesn’t support options -c and -a. After removing these options image was captured sucessfully.

diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
index c1a1a4f..28ef51b 100755
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
@@ -197,7 +197,7 @@ beginUpload() {
             imgpart="$imagePath/$img"
             mkfifo /tmp/pigz1 >/dev/null 2>&1
             uploadFormat "/tmp/pigz1" "$imgpart"
-            partclone.imager -c -s "$hd" -O /tmp/pigz1 -N -f 1 -a0
+            partclone.dd -s "$hd" -O /tmp/pigz1 -N -f 1
             rm /tmp/pigz1 >/dev/null 2>&1
             clearScreen
             ;;

[george1421]

Cross linking posts, while the post subject lines don’t match the net conditions do where its unable to capture a LVM volume" https://forums.fogproject.org/topic/14077/fog-not-detecting-target-disks-correctly-dev-sda-vs-dev-xvda

Will you confirm that on the FOG server either in /images/dev/<mac_address> or in /images/<image_name> that the disk and partition in question is created but the value (size) of the created .img file is bytes in size?

After fully digesting what you posted here this is the same exact issue as the other thread. Thank you for providing a solution @Developers comments or commits?

I believe the -a0 was added for block alignment for dedup reasons. Its a parameter for pigz and not partclone. Leaving that value in shouldn’t impact imaging unless option -a0 has been removed from pigz compression. As for the -c parameter for partclone, we will need to look at the change logs to understand if/why that parameter was removed.

[Tom Elliott]

@george1421 the -a0 is partclone 0.3.12 argument to disable crc checksumming IIRC. However, the inits downloaded by default are still using 0.2.89 partclone which did not have this argument available. Simply replacing the funcs.sh into these inits will cause issues.

As for LVM imaging, for all @developers, the inits already have functions created that should enable imaging of LVM volumes. The functions exist but I’ve not tested them. I believe they’re in the bottom of funcs.sh and the actual imaging scripts would need to be modified to look for and capture/deploy these volumes. They’ve never been tested, so may need to be and be reworked as needed. The lvm functions have been in there since around 1.4.4 was released.

[Tom Elliott]

@george1421 et al,

https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh#L2298

This is where the LVM stuff resides

[Tom Elliott]

and when it was actually added, January 2017 yay

https://github.com/FOGProject/fogproject/commit/224147284ad83f525e2c198fb5fbaaf1484bc171

[george1421]

In researching this issue I’m finding that this bit of code needs some attention in general.
ref: https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload#L196

        dd)
            imgpart="$imagePath/$img"
            mkfifo /tmp/pigz1 >/dev/null 2>&1
            uploadFormat "/tmp/pigz1" "$imgpart"
            partclone.imager -c -s "$hd" -O /tmp/pigz1 -N -f 1 -a0
            rm /tmp/pigz1 >/dev/null 2>&1
            clearScreen
            ;;

Where I was getting tripped up was thinking that -a0 was related to the -aX0 command from junkhacker’s request here: https://forums.fogproject.org/topic/13206/the-future-of-partclone-and-therefore-fog-as-it-is/98

It took me a bit to figure out what this code is doing, its pretty ingenious on how its using the fifo file and still compressing the image before uploading.

I can’t test the current build for partclone where I’m at so I can’t confirm if the -a0 here is even valid. I’ll be in a better position tomorrow to test the current inits.

[shruggy]

I’m not sure, but I believe that partclone.imager does have the -c option. It’s probably -a0 that got it tripped. (I haven’t checked it though). But partclone.dd has neither -c nor -a. Despite partclone.dd -h happily listing the last option. This is from the current partclone.c source:

if CHKIMG
	static const char *sopt = "-hvd::L:s:f:CFiBz:Nn:";
#elif RESTORE
	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
#elif DD
	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
#else
	static const char *sopt = "-hvd::L:cx:brDo:O:s:f:RCFINiqWBz:E:a:k:Kn:Tt";
#endif

[shruggy]

OK, partclone.imager seems to not have the -c option either. My trick works when capturing just one partition (or the whole disk) in dd mode, but to capture only the LVM volume in dd mode while allowing other modes for the rest of the disk, we’ll probably need something like:

diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
index 5d6054f..050ace9 100644
--- a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
+++ b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
@@ -2049,6 +2049,13 @@ savePartition() {
     getPartType "$part"
     local ebrfilename=""
     local swapuuidfilename=""
+    local clone="-c"
+    local checksum_mode="-a0"
+    if [ "0x8e" == "$parttype" ]; then
+        fstype=imager
+       clone=
+       checksum_mode=
+    fi
     case $fstype in
         swap)
             echo " * Saving swap partition UUID"
@@ -2068,7 +2075,7 @@ savePartition() {
                     debugPause
                     imgpart="$imagePath/d${disk_number}p${part_number}.img"
                     uploadFormat "$fifoname" "$imgpart"
-                    partclone.$fstype -n "Storage Location $storage, Image name $img" -cs $part -O $fifoname -Nf 1 -a0
+                    partclone.$fstype -n "Storage Location $storage, Image name $img" $clone -s $part -O $fifoname -N -f1 $checksum_mode
                     exitcode=$?
                     case $exitcode in
                         0)

Will test it tomorrow.

[Quazz]

To clarify -a0 disables the checksum; junkhacker in the original thread made a small mistake with -aX0 as the X was a placeholder that needed to be replaced with either a 0 or 1 for it to work. It shouldn’t cause any issues and removing it will break compatibility with images made under older versions of partclone.

partclone.dd not having a in its short options is actually a bug in my opinion as per

https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L352 clearly listing several options that are for some reason not included in the sopt list for dd.

That said; for dd specifically it turns out we can ignore that potential bug and remove a0 as per https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L402 setting the checksum_mode to 0 anyway.

And you are right that c is not an option for dd (anymore); I’m guesssing it was removed since a raw copy is always a clone so it makes no sense to specify it.

All that said, I think proper handling of LVM as per what Tom said should be the way forward.

[Sebastian Roth]

@Tom-Elliott Thanks heaps for pointing out that LVM support was prepared for some time ago! Didn’t know that.

However, the inits downloaded by default are still using 0.2.89 partclone which did not have this argument available.

That’s only true for 1.5.7 but people installing dev-branch (as posted here) already use inits with partclone version 0.3.12 (see here).

@shruggy Though I am not exactly sure I would imagine that switching from partclone 0.2.89 to 0.3.12 broke LVM raw capture.

@Quazz said:

All that said, I think proper handling of LVM as per what Tom said should be the way forward.

I’d think so too! The code being prepared for LVM already will make things easier I suppose and we might even add this “feature” before the next release as it turns out that newer versions break LVM (raw!) capture.

[Sebastian Roth]

@Quazz @shruggy Ok, after a major debugging session I am fairly sure I found why partclone.imager doesn’t work anymore the way we are used to it. Find new details here: https://github.com/FOGProject/fos/issues/35

On the question why we use partclone.imager instead of partclone.dd - also discussed in the other forum topic but just to give you the full picture: Intially partclone only had the dd mode that would save sector-per-sector (dd style) images that you’d need to restore using partclone.dd as well. But as partclone.imager was added it made things for us a bit easier to simply use partclone.restore for all images.

[shruggy]

I’ve built a new (or rather old) init.xz. It’s here. I didn’t have time and wanted to get things done quickly, so as a stop-gap measure I just replaced partclone 0.3.12 with the old version 0.2.89 0.2.91. Besides, the image also has some other differences (that hopefully will speed up testing it):

1. Buildroot 2019.11
2. Processor architecture set to Westmere
3. Partclone at 0.2.89 0.2.91
4. Zstd 1.4.4 (it’s really fast just as release notes promised)
5. MDadm 4.1
6. GPTfdisk 1.0.4

The whole diff is somewhat misleading, but nevertheless can be found here.

With it, I captured the disk from my master (still using FOG 1.5.7.89) and successfully restored it to another machine.

As @Sebastian-Roth has already found the root of the problem it’s probably unneeded anymore, but after this I was planning to update partclone on the image to the latest git snapshot on the release branch (that reports itself as 0.2.91). Done.

[Sebastian Roth]

@shruggy Thanks for sharing your work here! You might even think about joining the team and working on the official builds if you are keen.

Just a few comments:

Buildroot 2019.11

We usually stick to the LTS versions for the official builds. Just FYI.

Processor architecture set to Westmere

Do you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

Partclone at 0.2.89

You might want to use the proper patch file for that version for the web UI status bar to work properly.

Zstd 1.4.4 (it’s really fast just as release notes promised)
MDadm 4.1
GPTfdisk 1.0.4

Nice, will take a look at those and update our official builds as well.

BR2_KERNEL_HEADERS_4_19=y

Using this will make your init fail to run with older versions of the Linux kernel. Not a problem for you I suppose but we try to keep compatibility as good as possible.

As @Sebastian-Roth has already found the root of the problem it’s probably unneeded anymore, but after this I planned to update partclone on the image to the latest git snapshot on the release branch (that reports itself as 0.2.91).

I didn’t know the release branch is still on going… I thought it would be 0.2.89 -> 0.3.x

[shruggy]

@Sebastian-Roth

Processor architecture set to Westmere

Do you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

It definitely wouldn’t: Westmere are Intel Core processors from year 2010 IIRC.

Partclone at 0.2.89

You might want to use the proper patch file for that version for the web UI status bar to work properly.

Yes, i found and applied it, just failed to mention (or rather did it very clumsily: that bit about misleading diff).

[george1421]

@Sebastian-Roth said in 1.5.7.89: partclone doesn't capture an image in dd mode: wrong options in fog.upload:

o you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

FWIW: I had to build a one off kernel to image a 486 for a fog admin. So even with the stock bzImage32 anything less than a 586 isn’t supported.

We really don’t have any data on if there is a performance boost from making the minimum processor requirements to a specific model. It would be an interesting test. But the kernel and the inits would need to be in sync.

My gut reaction would be that it would cause more help desk calls about incompatible processors than any performance boost we might get from using newer instruction calls. But as they say, without data its only an opinion.

[shruggy]

Uploaded new version of the FOS image in my post: replaced partclone 0.2.89 with partclone 0.2.91 (aka the latest commit in release branch).

• checks for image version (currently 0001) and displays it on progress screen
• updates translations
• fixes compile issue on CentOS 6.8
• fixes --disable-* options behavior for configure
• updates Makefile
• additional checks for malloc as well as different sizes recorded in an image (used blocks, total blocks, device size)

[shruggy]

I tested and uploaded another version: this time it’s the latest Partclone with the fix suggested by @Sebastian-Roth, i.e.

diff --git a/src/ddclone.c b/src/ddclone.c
index 04f8775..cab5386 100644
--- a/src/ddclone.c
+++ b/src/ddclone.c
@@ -30,7 +30,9 @@ void read_super_blocks(char* device, file_system_info* fs_info)
        }
        strncpy(fs_info->fs, raw_MAGIC, FS_MAGIC_SIZE);
        fs_info->block_size  = PART_SECTOR_SIZE;
-       //fs_info->device_size = get_partition_size(&src);
+#ifdef IMG
+       fs_info->device_size = get_partition_size(&src);
+#endif
        fs_info->totalblock  = fs_info->device_size / PART_SECTOR_SIZE;
        fs_info->usedblocks  = fs_info->device_size / PART_SECTOR_SIZE;
        close(src);

The symbol IMG is defined in Makefile when building partclone.imager and prevents this change from being built into partclone.dd. For now, I’ve successfully captured a disk from one machine and restored it to another one. After the coming Microsoft Patch Day (probably over the next weekend) I am planning to capture another disk image with this and deploy it to my pool in multi-cast mode.

This time, I also built the 64-bit kernel 4.19.94 and used it together with this image. It’s here.

[Sebastian Roth]

Marking as fixed as we have this added to dev-branch and it will be in the next release.