• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    1.5.7.89: partclone doesn't capture an image in dd mode: wrong options in fog.upload

    Scheduled Pinned Locked Moved Solved
    FOG Problems
    5
    18
    1.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      shruggy
      last edited by shruggy

      After upgrade to the latest dev-branch, partclone stopped to capture images in dd mode. My setup:
      dual-boot Windows 10 / CentOS 6 (LVM).
      lsblk -f output:

      NAME   FSTYPE      LABEL             UUID
      sda                                                                                     
      |-sda1 ntfs        System-reserviert 6A1494EA1494BB11                                   
      |-sda2 ntfs                          4C5E04DB5E04BFA6                                   
      |-sda3 ext2                          4aef9246-b9cb-41b2-889f-2cd08fc5e966               
      `-sda4 LVM2_member                   gYEf27-xHDD-5oCl-qjHi-Uyvy-ScAv-BPAS86
      

      lsblk output:

      NAME   MAJ:MIN RM   SIZE RO TYPE
      sda      8:0    0   477G  0 disk
      |-sda1   8:1    0   500M  0 part
      |-sda2   8:2    0   238G  0 part
      |-sda3   8:3    0 105.9M  0 part
      `-sda4   8:4    0   238G  0 part
      

      The NTFS partitions get captured normally, so does the Ext2 partition, but when it comes to LVM (/dev/sda4), the partclone ends immediately without any error with the resulting size of partition image being almost zero. Almost because of compression overhead (70 bytes or so), but partclone.info reports the size of the image as zero.
      /var/log/partclone.log (after doing debug capture):

      Partclone v0.3.13 http://partclone.org
      Starting to clone device (/dev/sda) to image (/tmp/pigz1)
      UID is root.
      source=/dev/sda, target=/tmp/pigz1
      open source file/device /dev/sda
      open target file/device /tmp/pigz1
      Initiate image options - version 0002
      Initial image hdr - get Super Block from partition
      Reading Super Block
      open source file/device /dev/sda
      0 blocks per checksum
      memory needed: 2097152 bytes
      bitmap 0 bytes, blocks 2*1048576 bytes, checksum 0 bytes
      initial main bitmap pointer 0x522d20
      Initial image hdr - read bitmap table
      Calculating bitmap... Please wait...
      check main bitmap pointer 0x522d20
      Writing super block and bitmap...
      io_all: write 110, 0 left.
      io_all: write 4, 0 left.
      done!
      print image information
      MODE: clone
      DEBUG: 3
      SOURCE: /dev/sda
      TARGET: /tmp/pigz1
      OVERWRITE: 1
      RESCUE: 0
      CHECK: 1
      QUIET: 0
      FRESH: 1
      FORCE: 0
      BTFILES: 0
      NCURSES: 0
      OFFSET DOMAIN: 0x0
      CHECKSUM: NONE
      CS SIZE: 0
      BLOCKS/CS: 0
      NOTE:
      File system:  raw
      Device size:    0 Byte = 0 Blocks
      Space in use:   0 Byte = 0 Blocks
      Free Space:     0 Byte = 0 Blocks
      Block size:   512 Byte
      Initial Progress bar
      #
      Buffer capacity = 2048, Blocks per cs = 0
      #
      Total block 0
      start backup data...
      Syncing... OK!
      Partclone successfully cloned the device (/dev/sda) to the image (/tmp/pigz1)
      

      What I tried:

      1. rebuilt FOS with the latest partclone (that’s why the log reports partclone 0.3.13):
      diff --git a/Buildroot/package/partclone/partclone.mk b/Buildroot/package/partclone/partclone.mk
      index 07c2022..3035cf2 100644
      --- a/Buildroot/package/partclone/partclone.mk
      +++ b/Buildroot/package/partclone/partclone.mk
      @@ -4,9 +4,9 @@
       #
       ################################################################################
      
      -PARTCLONE_VERSION = 0.3.12
      +PARTCLONE_VERSION = 58d138da6e0f473acd80a7c9b8544d104377d50f
       PARTCLONE_SOURCE = partclone-$(PARTCLONE_VERSION).tar.gz
      -PARTCLONE_SITE = http://partclone.nchc.org.tw/download/testing
      +PARTCLONE_SITE = $(call github,Thomas-Tsai,partclone,$(PARTCLONE_VERSION))
       PARTCLONE_INSTALL_STAGING = YES
       PARTCLONE_AUTORECONF = YES
       PARTCLONE_DEPENDENCIES += attr e2fsprogs libgcrypt lzo xz zlib xfsprogs ncurses host-pkgconf
      
      1. changed partclone.imager to partclone.dd in /bin/fog.upload
        Here I got the message from partclone.dd that it doesn’t support options -c and -a. After removing these options image was captured sucessfully.
      diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
      index c1a1a4f..28ef51b 100755
      --- a/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
      +++ b/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload
      @@ -197,7 +197,7 @@ beginUpload() {
                   imgpart="$imagePath/$img"
                   mkfifo /tmp/pigz1 >/dev/null 2>&1
                   uploadFormat "/tmp/pigz1" "$imgpart"
      -            partclone.imager -c -s "$hd" -O /tmp/pigz1 -N -f 1 -a0
      +            partclone.dd -s "$hd" -O /tmp/pigz1 -N -f 1
                   rm /tmp/pigz1 >/dev/null 2>&1
                   clearScreen
                   ;;
      
      1 Reply Last reply Reply Quote 0
      • george1421G
        george1421 Moderator
        last edited by george1421

        Cross linking posts, while the post subject lines don’t match the net conditions do where its unable to capture a LVM volume" https://forums.fogproject.org/topic/14077/fog-not-detecting-target-disks-correctly-dev-sda-vs-dev-xvda

        Will you confirm that on the FOG server either in /images/dev/<mac_address> or in /images/<image_name> that the disk and partition in question is created but the value (size) of the created .img file is bytes in size?

        After fully digesting what you posted here this is the same exact issue as the other thread. Thank you for providing a solution @Developers comments or commits?

        I believe the -a0 was added for block alignment for dedup reasons. Its a parameter for pigz and not partclone. Leaving that value in shouldn’t impact imaging unless option -a0 has been removed from pigz compression. As for the -c parameter for partclone, we will need to look at the change logs to understand if/why that parameter was removed.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        Tom ElliottT 2 Replies Last reply Reply Quote 0
        • Tom ElliottT
          Tom Elliott @george1421
          last edited by

          @george1421 the -a0 is partclone 0.3.12 argument to disable crc checksumming IIRC. However, the inits downloaded by default are still using 0.2.89 partclone which did not have this argument available. Simply replacing the funcs.sh into these inits will cause issues.

          As for LVM imaging, for all @developers, the inits already have functions created that should enable imaging of LVM volumes. The functions exist but I’ve not tested them. I believe they’re in the bottom of funcs.sh and the actual imaging scripts would need to be modified to look for and capture/deploy these volumes. They’ve never been tested, so may need to be and be reworked as needed. The lvm functions have been in there since around 1.4.4 was released.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

          1 Reply Last reply Reply Quote 0
          • Tom ElliottT
            Tom Elliott @george1421
            last edited by

            @george1421 et al,

            https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh#L2298

            This is where the LVM stuff resides

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            Tom ElliottT 1 Reply Last reply Reply Quote 0
            • Tom ElliottT
              Tom Elliott @Tom Elliott
              last edited by

              and when it was actually added, January 2017 yay

              https://github.com/FOGProject/fogproject/commit/224147284ad83f525e2c198fb5fbaaf1484bc171

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              1 Reply Last reply Reply Quote 0
              • george1421G
                george1421 Moderator
                last edited by george1421

                In researching this issue I’m finding that this bit of code needs some attention in general.
                ref: https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload#L196

                        dd)
                            imgpart="$imagePath/$img"
                            mkfifo /tmp/pigz1 >/dev/null 2>&1
                            uploadFormat "/tmp/pigz1" "$imgpart"
                            partclone.imager -c -s "$hd" -O /tmp/pigz1 -N -f 1 -a0
                            rm /tmp/pigz1 >/dev/null 2>&1
                            clearScreen
                            ;;
                

                Where I was getting tripped up was thinking that -a0 was related to the -aX0 command from junkhacker’s request here: https://forums.fogproject.org/topic/13206/the-future-of-partclone-and-therefore-fog-as-it-is/98

                It took me a bit to figure out what this code is doing, its pretty ingenious on how its using the fifo file and still compressing the image before uploading.

                I can’t test the current build for partclone where I’m at so I can’t confirm if the -a0 here is even valid. I’ll be in a better position tomorrow to test the current inits.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                1 Reply Last reply Reply Quote 0
                • S
                  shruggy
                  last edited by

                  I’m not sure, but I believe that partclone.imager does have the -c option. It’s probably -a0 that got it tripped. (I haven’t checked it though). But partclone.dd has neither -c nor -a. Despite partclone.dd -h happily listing the last option. This is from the current partclone.c source:

                  if CHKIMG
                  	static const char *sopt = "-hvd::L:s:f:CFiBz:Nn:";
                  #elif RESTORE
                  	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
                  #elif DD
                  	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
                  #else
                  	static const char *sopt = "-hvd::L:cx:brDo:O:s:f:RCFINiqWBz:E:a:k:Kn:Tt";
                  #endif
                  
                  1 Reply Last reply Reply Quote 0
                  • S
                    shruggy
                    last edited by shruggy

                    OK, partclone.imager seems to not have the -c option either. My trick works when capturing just one partition (or the whole disk) in dd mode, but to capture only the LVM volume in dd mode while allowing other modes for the rest of the disk, we’ll probably need something like:

                    diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
                    index 5d6054f..050ace9 100644
                    --- a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
                    +++ b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
                    @@ -2049,6 +2049,13 @@ savePartition() {
                         getPartType "$part"
                         local ebrfilename=""
                         local swapuuidfilename=""
                    +    local clone="-c"
                    +    local checksum_mode="-a0"
                    +    if [ "0x8e" == "$parttype" ]; then
                    +        fstype=imager
                    +       clone=
                    +       checksum_mode=
                    +    fi
                         case $fstype in
                             swap)
                                 echo " * Saving swap partition UUID"
                    @@ -2068,7 +2075,7 @@ savePartition() {
                                         debugPause
                                         imgpart="$imagePath/d${disk_number}p${part_number}.img"
                                         uploadFormat "$fifoname" "$imgpart"
                    -                    partclone.$fstype -n "Storage Location $storage, Image name $img" -cs $part -O $fifoname -Nf 1 -a0
                    +                    partclone.$fstype -n "Storage Location $storage, Image name $img" $clone -s $part -O $fifoname -N -f1 $checksum_mode
                                         exitcode=$?
                                         case $exitcode in
                                             0)
                    

                    Will test it tomorrow.

                    1 Reply Last reply Reply Quote 0
                    • Q
                      Quazz Moderator
                      last edited by Quazz

                      To clarify -a0 disables the checksum; junkhacker in the original thread made a small mistake with -aX0 as the X was a placeholder that needed to be replaced with either a 0 or 1 for it to work. It shouldn’t cause any issues and removing it will break compatibility with images made under older versions of partclone.

                      partclone.dd not having a in its short options is actually a bug in my opinion as per

                      https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L352 clearly listing several options that are for some reason not included in the sopt list for dd.

                      That said; for dd specifically it turns out we can ignore that potential bug and remove a0 as per https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L402 setting the checksum_mode to 0 anyway.

                      And you are right that c is not an option for dd (anymore); I’m guesssing it was removed since a raw copy is always a clone so it makes no sense to specify it.

                      All that said, I think proper handling of LVM as per what Tom said should be the way forward.

                      1 Reply Last reply Reply Quote 1
                      • S
                        Sebastian Roth Moderator
                        last edited by

                        @Tom-Elliott Thanks heaps for pointing out that LVM support was prepared for some time ago! Didn’t know that.

                        However, the inits downloaded by default are still using 0.2.89 partclone which did not have this argument available.

                        That’s only true for 1.5.7 but people installing dev-branch (as posted here) already use inits with partclone version 0.3.12 (see here).

                        @shruggy Though I am not exactly sure I would imagine that switching from partclone 0.2.89 to 0.3.12 broke LVM raw capture.

                        @Quazz said:

                        All that said, I think proper handling of LVM as per what Tom said should be the way forward.

                        I’d think so too! The code being prepared for LVM already will make things easier I suppose and we might even add this “feature” before the next release as it turns out that newer versions break LVM (raw!) capture.

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        1 Reply Last reply Reply Quote 0
                        • S
                          Sebastian Roth Moderator
                          last edited by

                          @Quazz @shruggy Ok, after a major debugging session I am fairly sure I found why partclone.imager doesn’t work anymore the way we are used to it. Find new details here: https://github.com/FOGProject/fos/issues/35

                          On the question why we use partclone.imager instead of partclone.dd - also discussed in the other forum topic but just to give you the full picture: Intially partclone only had the dd mode that would save sector-per-sector (dd style) images that you’d need to restore using partclone.dd as well. But as partclone.imager was added it made things for us a bit easier to simply use partclone.restore for all images.

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          1 Reply Last reply Reply Quote 1
                          • S
                            shruggy
                            last edited by shruggy

                            I’ve built a new (or rather old) init.xz. It’s here. I didn’t have time and wanted to get things done quickly, so as a stop-gap measure I just replaced partclone 0.3.12 with the old version 0.2.89 0.2.91. Besides, the image also has some other differences (that hopefully will speed up testing it):

                            1. Buildroot 2019.11
                            2. Processor architecture set to Westmere
                            3. Partclone at 0.2.89 0.2.91
                            4. Zstd 1.4.4 (it’s really fast just as release notes promised)
                            5. MDadm 4.1
                            6. GPTfdisk 1.0.4

                            The whole diff is somewhat misleading, but nevertheless can be found here.

                            With it, I captured the disk from my master (still using FOG 1.5.7.89) and successfully restored it to another machine.

                            As @Sebastian-Roth has already found the root of the problem it’s probably unneeded anymore, but after this I was planning to update partclone on the image to the latest git snapshot on the release branch (that reports itself as 0.2.91). Done.

                            1 Reply Last reply Reply Quote 0
                            • S
                              Sebastian Roth Moderator
                              last edited by

                              @shruggy Thanks for sharing your work here! You might even think about joining the team and working on the official builds if you are keen.

                              Just a few comments:

                              Buildroot 2019.11

                              We usually stick to the LTS versions for the official builds. Just FYI.

                              Processor architecture set to Westmere

                              Do you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

                              Partclone at 0.2.89

                              You might want to use the proper patch file for that version for the web UI status bar to work properly.

                              Zstd 1.4.4 (it’s really fast just as release notes promised)
                              MDadm 4.1
                              GPTfdisk 1.0.4

                              Nice, will take a look at those and update our official builds as well.

                              BR2_KERNEL_HEADERS_4_19=y

                              Using this will make your init fail to run with older versions of the Linux kernel. Not a problem for you I suppose but we try to keep compatibility as good as possible.

                              As @Sebastian-Roth has already found the root of the problem it’s probably unneeded anymore, but after this I planned to update partclone on the image to the latest git snapshot on the release branch (that reports itself as 0.2.91).

                              I didn’t know the release branch is still on going… I thought it would be 0.2.89 -> 0.3.x

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              S george1421G 2 Replies Last reply Reply Quote 0
                              • S
                                shruggy @Sebastian Roth
                                last edited by shruggy

                                @Sebastian-Roth

                                Processor architecture set to Westmere

                                Do you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

                                It definitely wouldn’t: Westmere are Intel Core processors from year 2010 IIRC.

                                Partclone at 0.2.89

                                You might want to use the proper patch file for that version for the web UI status bar to work properly.

                                Yes, i found and applied it, just failed to mention (or rather did it very clumsily: that bit about misleading diff).

                                1 Reply Last reply Reply Quote 0
                                • george1421G
                                  george1421 Moderator @Sebastian Roth
                                  last edited by george1421

                                  @Sebastian-Roth said in 1.5.7.89: partclone doesn't capture an image in dd mode: wrong options in fog.upload:

                                  o you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

                                  FWIW: I had to build a one off kernel to image a 486 for a fog admin. So even with the stock bzImage32 anything less than a 586 isn’t supported.

                                  We really don’t have any data on if there is a performance boost from making the minimum processor requirements to a specific model. It would be an interesting test. But the kernel and the inits would need to be in sync.

                                  My gut reaction would be that it would cause more help desk calls about incompatible processors than any performance boost we might get from using newer instruction calls. But as they say, without data its only an opinion.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    shruggy
                                    last edited by

                                    Uploaded new version of the FOS image in my post: replaced partclone 0.2.89 with partclone 0.2.91 (aka the latest commit in release branch).

                                    • checks for image version (currently 0001) and displays it on progress screen
                                    • updates translations
                                    • fixes compile issue on CentOS 6.8
                                    • fixes --disable-* options behavior for configure
                                    • updates Makefile
                                    • additional checks for malloc as well as different sizes recorded in an image (used blocks, total blocks, device size)
                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      shruggy
                                      last edited by shruggy

                                      I tested and uploaded another version: this time it’s the latest Partclone with the fix suggested by @Sebastian-Roth, i.e.

                                      diff --git a/src/ddclone.c b/src/ddclone.c
                                      index 04f8775..cab5386 100644
                                      --- a/src/ddclone.c
                                      +++ b/src/ddclone.c
                                      @@ -30,7 +30,9 @@ void read_super_blocks(char* device, file_system_info* fs_info)
                                              }
                                              strncpy(fs_info->fs, raw_MAGIC, FS_MAGIC_SIZE);
                                              fs_info->block_size  = PART_SECTOR_SIZE;
                                      -       //fs_info->device_size = get_partition_size(&src);
                                      +#ifdef IMG
                                      +       fs_info->device_size = get_partition_size(&src);
                                      +#endif
                                              fs_info->totalblock  = fs_info->device_size / PART_SECTOR_SIZE;
                                              fs_info->usedblocks  = fs_info->device_size / PART_SECTOR_SIZE;
                                              close(src);
                                      

                                      The symbol IMG is defined in Makefile when building partclone.imager and prevents this change from being built into partclone.dd. For now, I’ve successfully captured a disk from one machine and restored it to another one. After the coming Microsoft Patch Day (probably over the next weekend) I am planning to capture another disk image with this and deploy it to my pool in multi-cast mode.

                                      This time, I also built the 64-bit kernel 4.19.94 and used it together with this image. It’s here.

                                      S 1 Reply Last reply Reply Quote 2
                                      • S
                                        Sebastian Roth Moderator
                                        last edited by

                                        Marking as fixed as we have this added to dev-branch and it will be in the next release.

                                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                        1 Reply Last reply Reply Quote 0
                                        • 1 / 1
                                        • First post
                                          Last post

                                        155

                                        Online

                                        12.0k

                                        Users

                                        17.3k

                                        Topics

                                        155.2k

                                        Posts
                                        Copyright © 2012-2024 FOG Project