• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Dedupe storage - how to best bypass pigz packaging?

    Scheduled Pinned Locked Moved Unsolved
    FOG Problems
    5
    16
    3.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Tom ElliottT
      Tom Elliott
      last edited by

      Prototype is up and appears to be working properly. With this new change it should theoretically be possible to use clonezilla images within fog, once files are named into fog formats. Same goes in reverse, fog images with uncompressed format should be able move into clonezilla provided the images are renamed to clonezilla naming standards.

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      george1421G 1 Reply Last reply Reply Quote 2
      • george1421G
        george1421 Moderator @Tom Elliott
        last edited by

        @Tom-Elliott Just so I’m clear, this feature is only available with the latest release of FOG (1.3.5.rc5)?

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

        Tom ElliottT 1 Reply Last reply Reply Quote 0
        • H
          HROAdmin26
          last edited by HROAdmin26

          @Tom-Elliott Thanks Tom! I will test out the funcs.sh change to see how the upload results change.

          @george1421 Yes, a clonezilla image directly uploaded will dedupe (with no compression.) The file is chunked into pieces and deduped based on those chunks. The art is to match the dedupe chunk size with the data inside the image and to match the chunk boundaries between the algorithm and the incoming data.

          1 Reply Last reply Reply Quote 0
          • Tom ElliottT
            Tom Elliott @george1421
            last edited by

            @george1421 Correct. Really the working branch, it will be available for re-installs of rc5 but not in the GUI.

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            1 Reply Last reply Reply Quote 0
            • Tom ElliottT
              Tom Elliott
              last edited by

              1.3.5 RC 6 has been released and should have this ‘uncompressed’ capability coded more properly for it.

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

              Jaymes DriverJ 1 Reply Last reply Reply Quote 1
              • Jaymes DriverJ
                Jaymes Driver Developer @Tom Elliott
                last edited by

                @Tom-Elliott Excitedly downloads new RC

                WARNING TO USERS: My comments are written completely devoid of emotion, do not mistake my concise to the point manner as a personal insult or attack.

                1 Reply Last reply Reply Quote 1
                • JunkhackerJ
                  Junkhacker Developer
                  last edited by

                  sorry i didn’t see this thread earlier, but i have experimented with dedup of uncompressed fog images on a zfs filesystem to see if it was worth it. i saw far less gains than when using compression. but i really can’t say i’m highly experienced in dedup, so maybe i did something wrong. let us know how your experiments go

                  signature:
                  Junkhacker
                  We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                  george1421G 1 Reply Last reply Reply Quote 1
                  • george1421G
                    george1421 Moderator @Junkhacker
                    last edited by

                    @Junkhacker I to am interested in seeing how the dedup rate compares to the same file compressed at a level 6. If for just information only. It would be interesting to know.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                    1 Reply Last reply Reply Quote 2
                    • Tom ElliottT
                      Tom Elliott
                      last edited by

                      I think de-duplication from multiple images (like 10 images) would be much more suitable than the same 10 images with compression. But again, this means it will be seen when you are dealing with multiple images. For one or two images it’s probably not going to be worth the gain.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                      JunkhackerJ 1 Reply Last reply Reply Quote 0
                      • JunkhackerJ
                        Junkhacker Developer @Tom Elliott
                        last edited by

                        @Tom-Elliott it was quite a while ago i did my testing (and of course i didn’t actually document anything…) but i was working with about 3 or 5 images seeing how much they shared so i could estimate from there, and it wasn’t good. very little duplication detected in spite of the obvious duplication that was taking place.
                        someone who knows what they’re doing might have much different results

                        signature:
                        Junkhacker
                        We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                        1 Reply Last reply Reply Quote 0
                        • 1 / 1
                        • First post
                          Last post

                        215

                        Online

                        12.0k

                        Users

                        17.3k

                        Topics

                        155.2k

                        Posts
                        Copyright © 2012-2024 FOG Project