• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

ZSTD Compression

Scheduled Pinned Locked Moved Solved
Feature Request
13
88
44.8k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J
    Junkhacker Developer @Tom Elliott
    last edited by Junkhacker Feb 20, 2017, 1:57 PM Feb 20, 2017, 7:56 PM

    @Tom-Elliott i was just performing some testing at zstandard -11 to compare to pigz -6
    image size went from 16.4GB gz to 14.6GB zstdmt
    capture time 14 minutes 15 seconds pigz to 11 minutes 4 seconds zstdmt
    deploy time 3 minutes 33 seconds pigz to 2 minutes 38 seconds zstdmt

    signature:
    Junkhacker
    We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

    1 Reply Last reply Reply Quote 0
    • W
      Wayne Workman
      last edited by Feb 20, 2017, 8:01 PM

      geeze that is flippin fast… ok I’m sold.

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
      Daily Clean Installation Results:
      https://fogtesting.fogproject.us/
      FOG Reporting:
      https://fog-external-reporting-results.fogproject.us/

      1 Reply Last reply Reply Quote 0
      • J
        Junkhacker Developer
        last edited by Feb 20, 2017, 10:16 PM

        i have not been able to get “ultra” levels (>=20) tasks to work without crashing. anyone else seeing the same?

        signature:
        Junkhacker
        We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

        T C 2 Replies Last reply Feb 21, 2017, 2:17 AM Reply Quote 1
        • ?
          A Former User
          last edited by Feb 21, 2017, 1:29 AM

          Anyone tried like a 40 GB image? That’s in the range where I’d like to see a comparison in deployment time.

          G J 2 Replies Last reply Feb 21, 2017, 1:32 AM Reply Quote 0
          • G
            george1421 Moderator @A Former User
            last edited by Feb 21, 2017, 1:32 AM

            @loosus456 I would say that a 40GB disk image is uncommon. If you have a source, then I might suggest that you update to the latest RC release and test the results.

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

            ? 1 Reply Last reply Feb 21, 2017, 1:42 AM Reply Quote 0
            • ?
              A Former User @george1421
              last edited by Feb 21, 2017, 1:42 AM

              @george1421 40 GB is uncommon? Uh what? Are you guys deploying XP or something?

              T W 2 Replies Last reply Feb 21, 2017, 2:18 AM Reply Quote 1
              • T
                Tom Elliott @Junkhacker
                last edited by Feb 21, 2017, 2:17 AM

                @Junkhacker I’m seeing exactly the same problem, and It feels like it’s writing the data and filling the RAM space too quickly. It’s doing so far faster than it can compress it, and overruns the memory space that partclone is using. (Just my theory).

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                1 Reply Last reply Reply Quote 0
                • T
                  Tom Elliott @A Former User
                  last edited by Feb 21, 2017, 2:18 AM

                  @loosus456 My base images with all software had a disk usage of 48gb at it’s largest, but that was EVERYTHING I could put on. My “average” was about 25gb.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  ? F 2 Replies Last reply Feb 21, 2017, 2:55 AM Reply Quote 3
                  • C
                    compman @Junkhacker
                    last edited by Feb 21, 2017, 2:30 AM

                    @Junkhacker “ultra” compression ratio (>=20) use a lot of memory.
                    So, either stick to “normal” compression levels (<= 19), or limit the number of threads, to ensure it doesn’t consume too much memory.

                    1 Reply Last reply Reply Quote 0
                    • ?
                      A Former User @Tom Elliott
                      last edited by Feb 21, 2017, 2:55 AM

                      @Tom-Elliott We have images with Autodesk and Solidworks near 100 GB.

                      T 1 Reply Last reply Feb 21, 2017, 2:56 AM Reply Quote 2
                      • T
                        Tom Elliott @A Former User
                        last edited by Feb 21, 2017, 2:56 AM

                        @loosus456 I’m not disagreeing, I’m just saying what I imagine “most” people are working with (on average).

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        ? 1 Reply Last reply Feb 21, 2017, 3:04 AM Reply Quote 0
                        • ?
                          A Former User @Tom Elliott
                          last edited by A Former User Feb 20, 2017, 9:04 PM Feb 21, 2017, 3:04 AM

                          @Tom-Elliott I dunno. Most educational professionals I work with have had similar sizes.

                          1 Reply Last reply Reply Quote 0
                          • W
                            Wayne Workman @A Former User
                            last edited by Wayne Workman Feb 20, 2017, 9:57 PM Feb 21, 2017, 3:54 AM

                            @loosus456 said in ZSTD Compression:

                            40 GB is uncommon?

                            My Win10 images were around 20GB. Linux images are even smaller, 3 or 4 GB - and that’s all I deploy anymore, just all Linux.

                            Understand when I build an image - it’s lean. No flab. I literally throw out the manufacturer’s bloated crappy flabby unoptimized image and build my own from scratch. Not just for size & performance reasons but for security reasons too. Computer manufacturers don’t vet the bloatware they sign contracts for - to install into their images - many times these crappy bloated pieces of software are found to have vulnerabilities. But I make images lean. In Windows that means a vanilla installation & using device manager to install ONLY the drivers and no extra flab/bloatware crap. When I put MS Office on the image, I only installed the pieces that people used, not everything. The fat 10GB recovery partition gets thrown out, don’t need that. I also rebuilt all my images every summer - because if all you do is take an old image and patch it up, it gets bloated & slow.

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                            Daily Clean Installation Results:
                            https://fogtesting.fogproject.us/
                            FOG Reporting:
                            https://fog-external-reporting-results.fogproject.us/

                            1 Reply Last reply Reply Quote 0
                            • F
                              falko Moderator @Tom Elliott
                              last edited by falko Feb 21, 2017, 12:41 AM Feb 21, 2017, 6:33 AM

                              Tom-Elliott said in ZSTD Compression:

                              @loosus456 My base images with all software had a disk usage of 48gb at it’s largest, but that was EVERYTHING I could put on. My “average” was about 25gb.

                              I use a single win7 image at my place (nearly all software etc.), it’s just under 30gb. Will make some time to do some tests

                              1 Reply Last reply Reply Quote 0
                              • sudburrS
                                sudburr @Junkhacker
                                last edited by Feb 21, 2017, 4:03 PM

                                @Junkhacker The majority of my images are ~55 GiB compressed, 85-115 GiB deployed. And yah, Autodesk and Adobe products suck it up real good.

                                [ Standing in between extinction in the cold and explosive radiating growth ]

                                1 Reply Last reply Reply Quote 1
                                • J
                                  Junkhacker Developer @A Former User
                                  last edited by Junkhacker Feb 21, 2017, 4:20 PM Feb 21, 2017, 10:11 PM

                                  @loosus456 the image i was testing with here was 33.87 GB “on client”

                                  signature:
                                  Junkhacker
                                  We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                                  1 Reply Last reply Reply Quote 0
                                  • V
                                    VincentJ Moderator
                                    last edited by Feb 23, 2017, 8:23 PM

                                    So… Deploy with old fog, capture with new fog on a different image set to zstd 11

                                    I noticed on the upload, the screen still says:
                                    Starting to clone device (/dev/sda2) to image (/tmp/pigz1)

                                    The image is Multiple Partition Single Disk (non re-sizeable)

                                    On my old image it has two files… but the new one has three…

                                    I am pulling the image off the NAS now to unpack it and test it, but thought i would pitch in on what i’ve seen so far.

                                    svn revision 6066

                                    J 1 Reply Last reply Feb 23, 2017, 8:31 PM Reply Quote 0
                                    • J
                                      Junkhacker Developer @VincentJ
                                      last edited by Feb 23, 2017, 8:31 PM

                                      @VincentJ “/tmp/pigz1” is just the name of the fifo that the data is being piped into to be sent to compression. maybe we should rename it for purely aesthetic reasons, but that’s working as it should. was the previous image also Multiple Partition Single Disk (non re-sizeable)?

                                      signature:
                                      Junkhacker
                                      We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                                      V 1 Reply Last reply Feb 24, 2017, 4:21 AM Reply Quote 1
                                      • V
                                        VincentJ Moderator @Junkhacker
                                        last edited by Feb 24, 2017, 4:21 AM

                                        @Junkhacker Yes it was. All of my images are.

                                        1 Reply Last reply Reply Quote 0
                                        • JJ FullmerJ
                                          JJ Fullmer Testers
                                          last edited by Mar 15, 2017, 8:11 PM

                                          So I just updated to RC 15 and made a new image at zstd level 9. I was using gzip level 5 before. The image is still currently downloading, estimating the full time to be about 12 minutes. With gzip the download was taking around 3 minutes. This is a 6th gen i5 client and the fogserver is on a vm with 8 vcpus (intel xeon something something) and 16 GB ram. There is also a storage node on a older server with a little less power, but still imaged around 2-3 minutes.
                                          The size of the image uncompressed is ~15 GB and it is 7.5 GB compressed with zstd. My old image was ~18 GB and with gzip level 5 it compressed to 6.7 GB.

                                          So what did I do wrong? Is the difference between 9 and 11 really that substantial? Should I re-upload that image? Also split vs non split? I saw the post on the topic of it being better for multiple images, but are there any other considerations in regard to speed?

                                          Have you tried the FogApi powershell module? It's pretty cool IMHO
                                          https://github.com/darksidemilk/FogApi
                                          https://fogapi.readthedocs.io/en/latest/
                                          https://www.powershellgallery.com/packages/FogApi
                                          https://forums.fogproject.org/topic/12026/powershell-api-module

                                          T 1 Reply Last reply Mar 15, 2017, 8:12 PM Reply Quote 0
                                          • 1
                                          • 2
                                          • 3
                                          • 4
                                          • 5
                                          • 4 / 5
                                          • First post
                                            Last post

                                          161

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project