• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    The future of partclone and therefore FOG as it is

    Scheduled Pinned Locked Moved
    General
    6
    122
    54.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • george1421G
      george1421 Moderator @Junkhacker
      last edited by

      @Junkhacker Sorry I haven’t had time between traveling and project deadlines most everything else has got put on back burner. Both you and Quazz say they are working fine. I’m not sure why my images are not. I know at least one I tested with was created with 1.5.4 the rest may be 1.4.4, but I’d have to look at the image creation date to be sure. Everything looks like it deploys correctly with partclone, just when I try to mount the 😄 partition and copy the drivers over it says no space left on the device.

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

      JunkhackerJ Q 3 Replies Last reply Reply Quote 0
      • JunkhackerJ
        Junkhacker Developer @george1421
        last edited by

        @george1421 No problem. I was just hoping to make progress on this. I found this https://github.com/stevenshiau/clonezilla/issues/31 and wondered if it was related since it was from when clonezilla updated their partclone version.

        signature:
        Junkhacker
        We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

        1 Reply Last reply Reply Quote 0
        • Q
          Quazz Moderator @george1421
          last edited by

          @george1421 Well, potentially that was a resize bug in the inits if you’re using an NVME drive.

          Could also be the mount failing for some reason?

          Just some ideas.

          1 Reply Last reply Reply Quote 1
          • Q
            Quazz Moderator @george1421
            last edited by

            @george1421 Well, we did find a resize bug in the inits, particularily on fast disks.

            Might be time to revive this idea soon and test it out again.

            george1421G 1 Reply Last reply Reply Quote 1
            • george1421G
              george1421 Moderator @Quazz
              last edited by

              @Quazz Great idea.

              I’m rebuilding the inits with partclone 0.3.13 and zstd 1.4.0 using the FOG 1.5.7 base files for the inits. I suspect trouble the first time through compiling this since I’ve forgotten everything that I knew from before. Plus I need to update my test fog server to 1.5.7 so it will be a challenging morning to say the least.

              @Sebastian-Roth Are you still using buildroot v2019.02.1 for the inits?

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

              Q 1 Reply Last reply Reply Quote 0
              • Q
                Quazz Moderator @george1421
                last edited by

                @george1421 I don’t believe FOS github has the fix included yet at this point, though shouldn’t be too long.

                Still trying to figure out if/when/where the problem exactly started.

                As far as I could find it’s supposedly been around for ages (sfdisk - udev conflict), but isn’t necessarily triggered usually.

                1 Reply Last reply Reply Quote 0
                • S
                  Sebastian Roth Moderator
                  last edited by

                  @george1421 Yeah Quazz is right. I was in a bit of a rush to get some minor issues fixed when releasing the inits and did not push the changes to github yet. I will sort all this in a couple of hours though. Don’t rush into testing partclone yet as we might even think about moving to the latest buildroot in the next days too.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  george1421G 1 Reply Last reply Reply Quote 0
                  • george1421G
                    george1421 Moderator @Sebastian Roth
                    last edited by

                    @Sebastian-Roth OK I’ll hold off until I get the green light from you.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                    1 Reply Last reply Reply Quote 0
                    • Q
                      Quazz Moderator
                      last edited by

                      I’m preparing a build with Buildroot version 2019.02.4 and one with that Buildroot version + partclone 3.12, will test and post them on Monday probably.

                      george1421G 1 Reply Last reply Reply Quote 2
                      • george1421G
                        george1421 Moderator @Quazz
                        last edited by

                        @Quazz Here are the edits (not in diff format but george shorthand) that junkhacker recommended to take advantage of new features in pigz and zstd for data dedup systems

                        vi fog.upload
                        partclone.imager -aX0 -c -s "$hd" -O /tmp/pigz1 -N -f 1
                        
                        vi funcs.sh
                        pigz --rsyncable $PIGZ_COMP < $fifo | split -a 3 -d -b 200m - ${file}. &
                        pigz --rsyncable $PIGZ_COMP < $fifo > ${file}.000 &
                        
                        zstdmt --rsyncable -B128 --ultra $PIGZ_COMP < $fifo | split -a 3 -d -b 200m - ${file}. &
                        zstdmt --rsyncable -B128 --ultra $PIGZ_COMP < $fifo > ${file}.000 &
                        
                        partclone.$fstype -aX0 -n "Storage Location $storage        , Image name $img" -cs $part -O $fifoname -Nf 1
                        
                        remove --ignore_crc
                        
                        

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                        Q 1 Reply Last reply Reply Quote 2
                        • Q
                          Quazz Moderator @george1421
                          last edited by

                          @george1421 Thanks, I had already done some of that, got all of it in there now.

                          1 Reply Last reply Reply Quote 0
                          • S
                            Sebastian Roth Moderator
                            last edited by

                            @george1421 said in The future of partclone and therefore FOG as it is:

                            OK I’ll hold off until I get the green light from you.

                            Totally forgot to let you know that it’s been done - weeks ago already (ref).

                            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                            1 Reply Last reply Reply Quote 1
                            • Q
                              Quazz Moderator
                              last edited by Quazz

                              Okay, so I can’t believe I never noticed this, but -aX0 is an invalid parameter, it should be -a0

                              Source: https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c

                              		"    -aX  --checksum-mode=X  Checksum formula to use to add error detection\n"
                              		"                            where X:\n"
                              		"                            0: No checksum (no slowdown, smallest image)\n"
                              		"                            1: CRC32 (Fast to compute, basic detection)\n"
                              

                              Hopefully the source of the funky problem I was having, rebuilding now.

                              1 Reply Last reply Reply Quote 2
                              • Q
                                Quazz Moderator
                                last edited by Quazz

                                The error I was getting seems to be originating from zstdmt, it doesn’t have an rsyncable option in the included version. Would have thought that would be bundled by now, grrr.

                                george1421G 1 Reply Last reply Reply Quote 0
                                • george1421G
                                  george1421 Moderator @Quazz
                                  last edited by george1421

                                  @Quazz I think in my dev environment I change the BR package to download zstd 1.4.0. My build environment server had bad ssd, so I’m going to rebuild it. I can’t get to the files at the moment to confirm through.

                                  edit: Yes zstd 1.4.0
                                  zstd.mk: ZSTD_VERSION = v1.4.0

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                  Q 1 Reply Last reply Reply Quote 0
                                  • Q
                                    Quazz Moderator @george1421
                                    last edited by

                                    @george1421 I decided to go for 1.4.2 since it includes a couple of performance improvements and bug fixes.

                                    Took a while longer because I forgot to include the hash file.

                                    It’s finally compiling successfully, time for some more tests.

                                    1 Reply Last reply Reply Quote 2
                                    • Q
                                      Quazz Moderator
                                      last edited by Quazz

                                      Currently it fails to capture msftres partitions, throwing exit code 139 (presumably zstd’s error code). msftres is captured with partclone.imager

                                      It’s difficult to see what’s going on because the screen breaks up, but it seems to not recognize any size on this type of partition on partclone 3.12 which is a regression compared to 2.89

                                      edit: Adding a strategic debugPause tells me there is a segmentation fault on line 2041 in funcs.sh

                                      Even stranger is that it is talking about line 2053, yet for some reason mentions line 2041???

                                      Though not sure why this occurs…

                                      edit2: Seems to be down to the fact that it writes to /tmp/pigz1

                                      Direct writing to /images works fine

                                      edit3: Welp, it seems to come and go as it pleases, not sure what the actual cause is!

                                      edit4: My suspicion lies on B128 option being the culprit (initial test is promising), recompiling, will test tomorrow.

                                      1 Reply Last reply Reply Quote 1
                                      • Q
                                        Quazz Moderator
                                        last edited by Quazz

                                        Removing the -B128 option seems to allow it to capture, finally!

                                        64 bit init.xz Buildroot 2019.02.4 + ZSTD 1.4.2 + Partclone 0.3.12 (with APFS support)

                                        32 bit init.xz Buildroot 2019.02.4 + ZSTD 1.4.2 + Partclone 0.3.12 (with APFS support)

                                        Give it a whirl if you’re interested.

                                        Current status:

                                        • Specifying the -B128 option gives issues in certain scenarios (special partitions/raw/very small ones???), so we can’t reliably use that.

                                        • In order to use --rsyncable, zstd had to be updated to a minimum of 1.3.8 (chosen the latest version of 1.4.2 to include performance improvements and bug fixes)

                                        • Minor Buildroot update means config doesn’t have to be updated, it’s just bugfixes

                                        • Added APFS support so that we can offer some better support for newer Macs (if they decide to PXE boot that is) (potentially gptfdisk package should be updated to 1.0.4 (adds typecodes for APFS and others), haven’t tested anything in this direction!)

                                        george1421G 1 Reply Last reply Reply Quote 3
                                        • Q
                                          Quazz Moderator
                                          last edited by Quazz

                                          I’m also wondering if we should look into updating other packages that FOG either overrides or provides manually. (eg testdisk is on 6.14 as opposed to 7.1 and even pigz is at 2.3.4 instead of 2.4 (although pigz seems to have introduced potentially backwards incompatible changes))

                                          1 Reply Last reply Reply Quote 0
                                          • george1421G
                                            george1421 Moderator @Quazz
                                            last edited by

                                            @Quazz Great news that you got it to compile. Also APFS support was a current topic wondering if it was going to be supported. So good idea to add that support. To make use of that will the web gui need to be updated to for a different operating system type?

                                            I’ll download the inits a bit later today to try them in my environment. I’m still running FOG 1.5.5 FWIW.

                                            As for updating other packages… (understand this is just my opinion) You installed buildroot 2019.02.1 which is the same as what the developers are using. Every time you deviate from the standard buildroot package you run the risk of introducing unexpected bugs. I would say unless there is a compelling reason (fog needs a feature of the update package or to fix a bug fog is hitting) don’t upgrade packages at random. Let the buildroot devs do that leg work. I also saw that pigz had an updated version and I think I saw the incompatibility statements so I decided to not update that in my build. That was simply because pigz supported the resyncable flag that we were looking to implement. So I would recommend that unless we are trying to solve a problem, I would say hold off on updating any packages beyond the buildroot stream. I would rather want stable over new unneeded features.

                                            Great job on getting the inits compiled and working!! I’ll let you know how it works on my stuff.

                                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                            Q 1 Reply Last reply Reply Quote 1
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 5
                                            • 6
                                            • 7
                                            • 5 / 7
                                            • First post
                                              Last post

                                            143

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project