• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

1.5.7.89: partclone doesn't capture an image in dd mode: wrong options in fog.upload

Scheduled Pinned Locked Moved Solved
FOG Problems
5
18
1.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • G
    george1421 Moderator
    last edited by george1421 Jan 1, 2020, 6:12 PM Jan 1, 2020, 11:39 PM

    In researching this issue I’m finding that this bit of code needs some attention in general.
    ref: https://github.com/FOGProject/fos/blob/master/Buildroot/board/FOG/FOS/rootfs_overlay/bin/fog.upload#L196

            dd)
                imgpart="$imagePath/$img"
                mkfifo /tmp/pigz1 >/dev/null 2>&1
                uploadFormat "/tmp/pigz1" "$imgpart"
                partclone.imager -c -s "$hd" -O /tmp/pigz1 -N -f 1 -a0
                rm /tmp/pigz1 >/dev/null 2>&1
                clearScreen
                ;;
    

    Where I was getting tripped up was thinking that -a0 was related to the -aX0 command from junkhacker’s request here: https://forums.fogproject.org/topic/13206/the-future-of-partclone-and-therefore-fog-as-it-is/98

    It took me a bit to figure out what this code is doing, its pretty ingenious on how its using the fifo file and still compressing the image before uploading.

    I can’t test the current build for partclone where I’m at so I can’t confirm if the -a0 here is even valid. I’ll be in a better position tomorrow to test the current inits.

    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

    1 Reply Last reply Reply Quote 0
    • S
      shruggy
      last edited by Jan 1, 2020, 11:42 PM

      I’m not sure, but I believe that partclone.imager does have the -c option. It’s probably -a0 that got it tripped. (I haven’t checked it though). But partclone.dd has neither -c nor -a. Despite partclone.dd -h happily listing the last option. This is from the current partclone.c source:

      if CHKIMG
      	static const char *sopt = "-hvd::L:s:f:CFiBz:Nn:";
      #elif RESTORE
      	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
      #elif DD
      	static const char *sopt = "-hvd::L:o:O:s:f:CFINiqWBz:E:n:Tt";
      #else
      	static const char *sopt = "-hvd::L:cx:brDo:O:s:f:RCFINiqWBz:E:a:k:Kn:Tt";
      #endif
      
      1 Reply Last reply Reply Quote 0
      • S
        shruggy
        last edited by shruggy Jan 1, 2020, 7:32 PM Jan 2, 2020, 1:18 AM

        OK, partclone.imager seems to not have the -c option either. My trick works when capturing just one partition (or the whole disk) in dd mode, but to capture only the LVM volume in dd mode while allowing other modes for the rest of the disk, we’ll probably need something like:

        diff --git a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
        index 5d6054f..050ace9 100644
        --- a/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
        +++ b/Buildroot/board/FOG/FOS/rootfs_overlay/usr/share/fog/lib/funcs.sh
        @@ -2049,6 +2049,13 @@ savePartition() {
             getPartType "$part"
             local ebrfilename=""
             local swapuuidfilename=""
        +    local clone="-c"
        +    local checksum_mode="-a0"
        +    if [ "0x8e" == "$parttype" ]; then
        +        fstype=imager
        +       clone=
        +       checksum_mode=
        +    fi
             case $fstype in
                 swap)
                     echo " * Saving swap partition UUID"
        @@ -2068,7 +2075,7 @@ savePartition() {
                             debugPause
                             imgpart="$imagePath/d${disk_number}p${part_number}.img"
                             uploadFormat "$fifoname" "$imgpart"
        -                    partclone.$fstype -n "Storage Location $storage, Image name $img" -cs $part -O $fifoname -Nf 1 -a0
        +                    partclone.$fstype -n "Storage Location $storage, Image name $img" $clone -s $part -O $fifoname -N -f1 $checksum_mode
                             exitcode=$?
                             case $exitcode in
                                 0)
        

        Will test it tomorrow.

        1 Reply Last reply Reply Quote 0
        • Q
          Quazz Moderator
          last edited by Quazz Jan 2, 2020, 3:47 AM Jan 2, 2020, 8:47 AM

          To clarify -a0 disables the checksum; junkhacker in the original thread made a small mistake with -aX0 as the X was a placeholder that needed to be replaced with either a 0 or 1 for it to work. It shouldn’t cause any issues and removing it will break compatibility with images made under older versions of partclone.

          partclone.dd not having a in its short options is actually a bug in my opinion as per

          https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L352 clearly listing several options that are for some reason not included in the sopt list for dd.

          That said; for dd specifically it turns out we can ignore that potential bug and remove a0 as per https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L402 setting the checksum_mode to 0 anyway.

          And you are right that c is not an option for dd (anymore); I’m guesssing it was removed since a raw copy is always a clone so it makes no sense to specify it.

          All that said, I think proper handling of LVM as per what Tom said should be the way forward.

          1 Reply Last reply Reply Quote 1
          • S
            Sebastian Roth Moderator
            last edited by Jan 2, 2020, 3:26 PM

            @Tom-Elliott Thanks heaps for pointing out that LVM support was prepared for some time ago! Didn’t know that.

            However, the inits downloaded by default are still using 0.2.89 partclone which did not have this argument available.

            That’s only true for 1.5.7 but people installing dev-branch (as posted here) already use inits with partclone version 0.3.12 (see here).

            @shruggy Though I am not exactly sure I would imagine that switching from partclone 0.2.89 to 0.3.12 broke LVM raw capture.

            @Quazz said:

            All that said, I think proper handling of LVM as per what Tom said should be the way forward.

            I’d think so too! The code being prepared for LVM already will make things easier I suppose and we might even add this “feature” before the next release as it turns out that newer versions break LVM (raw!) capture.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            1 Reply Last reply Reply Quote 0
            • S
              Sebastian Roth Moderator
              last edited by Jan 4, 2020, 3:47 PM

              @Quazz @shruggy Ok, after a major debugging session I am fairly sure I found why partclone.imager doesn’t work anymore the way we are used to it. Find new details here: https://github.com/FOGProject/fos/issues/35

              On the question why we use partclone.imager instead of partclone.dd - also discussed in the other forum topic but just to give you the full picture: Intially partclone only had the dd mode that would save sector-per-sector (dd style) images that you’d need to restore using partclone.dd as well. But as partclone.imager was added it made things for us a bit easier to simply use partclone.restore for all images.

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              1 Reply Last reply Reply Quote 1
              • S
                shruggy
                last edited by shruggy Jan 4, 2020, 4:52 PM Jan 4, 2020, 5:41 PM

                I’ve built a new (or rather old) init.xz. It’s here. I didn’t have time and wanted to get things done quickly, so as a stop-gap measure I just replaced partclone 0.3.12 with the old version 0.2.89 0.2.91. Besides, the image also has some other differences (that hopefully will speed up testing it):

                1. Buildroot 2019.11
                2. Processor architecture set to Westmere
                3. Partclone at 0.2.89 0.2.91
                4. Zstd 1.4.4 (it’s really fast just as release notes promised)
                5. MDadm 4.1
                6. GPTfdisk 1.0.4

                The whole diff is somewhat misleading, but nevertheless can be found here.

                With it, I captured the disk from my master (still using FOG 1.5.7.89) and successfully restored it to another machine.

                As @Sebastian-Roth has already found the root of the problem it’s probably unneeded anymore, but after this I was planning to update partclone on the image to the latest git snapshot on the release branch (that reports itself as 0.2.91). Done.

                1 Reply Last reply Reply Quote 0
                • S
                  Sebastian Roth Moderator
                  last edited by Jan 4, 2020, 6:47 PM

                  @shruggy Thanks for sharing your work here! You might even think about joining the team and working on the official builds if you are keen.

                  Just a few comments:

                  Buildroot 2019.11

                  We usually stick to the LTS versions for the official builds. Just FYI.

                  Processor architecture set to Westmere

                  Do you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

                  Partclone at 0.2.89

                  You might want to use the proper patch file for that version for the web UI status bar to work properly.

                  Zstd 1.4.4 (it’s really fast just as release notes promised)
                  MDadm 4.1
                  GPTfdisk 1.0.4

                  Nice, will take a look at those and update our official builds as well.

                  BR2_KERNEL_HEADERS_4_19=y

                  Using this will make your init fail to run with older versions of the Linux kernel. Not a problem for you I suppose but we try to keep compatibility as good as possible.

                  As @Sebastian-Roth has already found the root of the problem it’s probably unneeded anymore, but after this I planned to update partclone on the image to the latest git snapshot on the release branch (that reports itself as 0.2.91).

                  I didn’t know the release branch is still on going… I thought it would be 0.2.89 -> 0.3.x

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  S G 2 Replies Last reply Jan 4, 2020, 6:55 PM Reply Quote 0
                  • S
                    shruggy @Sebastian Roth
                    last edited by shruggy Jan 4, 2020, 1:30 PM Jan 4, 2020, 6:55 PM

                    @Sebastian-Roth

                    Processor architecture set to Westmere

                    Do you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

                    It definitely wouldn’t: Westmere are Intel Core processors from year 2010 IIRC.

                    Partclone at 0.2.89

                    You might want to use the proper patch file for that version for the web UI status bar to work properly.

                    Yes, i found and applied it, just failed to mention (or rather did it very clumsily: that bit about misleading diff).

                    1 Reply Last reply Reply Quote 0
                    • G
                      george1421 Moderator @Sebastian Roth
                      last edited by george1421 Jan 4, 2020, 2:51 PM Jan 4, 2020, 8:48 PM

                      @Sebastian-Roth said in 1.5.7.89: partclone doesn't capture an image in dd mode: wrong options in fog.upload:

                      o you know if this would run on good old 486 PCs as well? I guess we still have users with fairly old hardware around.

                      FWIW: I had to build a one off kernel to image a 486 for a fog admin. So even with the stock bzImage32 anything less than a 586 isn’t supported.

                      We really don’t have any data on if there is a performance boost from making the minimum processor requirements to a specific model. It would be an interesting test. But the kernel and the inits would need to be in sync.

                      My gut reaction would be that it would cause more help desk calls about incompatible processors than any performance boost we might get from using newer instruction calls. But as they say, without data its only an opinion.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                      1 Reply Last reply Reply Quote 0
                      • S
                        shruggy
                        last edited by Jan 4, 2020, 11:28 PM

                        Uploaded new version of the FOS image in my post: replaced partclone 0.2.89 with partclone 0.2.91 (aka the latest commit in release branch).

                        • checks for image version (currently 0001) and displays it on progress screen
                        • updates translations
                        • fixes compile issue on CentOS 6.8
                        • fixes --disable-* options behavior for configure
                        • updates Makefile
                        • additional checks for malloc as well as different sizes recorded in an image (used blocks, total blocks, device size)
                        1 Reply Last reply Reply Quote 0
                        • S
                          shruggy
                          last edited by shruggy Jan 15, 2020, 4:28 AM Jan 11, 2020, 5:18 PM

                          I tested and uploaded another version: this time it’s the latest Partclone with the fix suggested by @Sebastian-Roth, i.e.

                          diff --git a/src/ddclone.c b/src/ddclone.c
                          index 04f8775..cab5386 100644
                          --- a/src/ddclone.c
                          +++ b/src/ddclone.c
                          @@ -30,7 +30,9 @@ void read_super_blocks(char* device, file_system_info* fs_info)
                                  }
                                  strncpy(fs_info->fs, raw_MAGIC, FS_MAGIC_SIZE);
                                  fs_info->block_size  = PART_SECTOR_SIZE;
                          -       //fs_info->device_size = get_partition_size(&src);
                          +#ifdef IMG
                          +       fs_info->device_size = get_partition_size(&src);
                          +#endif
                                  fs_info->totalblock  = fs_info->device_size / PART_SECTOR_SIZE;
                                  fs_info->usedblocks  = fs_info->device_size / PART_SECTOR_SIZE;
                                  close(src);
                          

                          The symbol IMG is defined in Makefile when building partclone.imager and prevents this change from being built into partclone.dd. For now, I’ve successfully captured a disk from one machine and restored it to another one. After the coming Microsoft Patch Day (probably over the next weekend) I am planning to capture another disk image with this and deploy it to my pool in multi-cast mode.

                          This time, I also built the 64-bit kernel 4.19.94 and used it together with this image. It’s here.

                          S 1 Reply Last reply Jan 25, 2020, 7:52 PM Reply Quote 2
                          • S
                            Sebastian Roth Moderator
                            last edited by Jan 27, 2020, 7:18 PM

                            Marking as fixed as we have this added to dev-branch and it will be in the next release.

                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                            1 Reply Last reply Reply Quote 0
                            • 1 / 1
                            • First post
                              Last post

                            145

                            Online

                            12.0k

                            Users

                            17.3k

                            Topics

                            155.2k

                            Posts
                            Copyright © 2012-2024 FOG Project