• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

ZSTD Compression

Scheduled Pinned Locked Moved Solved
Feature Request
13
88
45.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    sudburr @Junkhacker
    last edited by Feb 21, 2017, 4:03 PM

    @Junkhacker The majority of my images are ~55 GiB compressed, 85-115 GiB deployed. And yah, Autodesk and Adobe products suck it up real good.

    [ Standing in between extinction in the cold and explosive radiating growth ]

    1 Reply Last reply Reply Quote 1
    • J
      Junkhacker Developer @A Former User
      last edited by Junkhacker Feb 21, 2017, 4:20 PM Feb 21, 2017, 10:11 PM

      @loosus456 the image i was testing with here was 33.87 GB “on client”

      signature:
      Junkhacker
      We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

      1 Reply Last reply Reply Quote 0
      • V
        VincentJ Moderator
        last edited by Feb 23, 2017, 8:23 PM

        So… Deploy with old fog, capture with new fog on a different image set to zstd 11

        I noticed on the upload, the screen still says:
        Starting to clone device (/dev/sda2) to image (/tmp/pigz1)

        The image is Multiple Partition Single Disk (non re-sizeable)

        On my old image it has two files… but the new one has three…

        I am pulling the image off the NAS now to unpack it and test it, but thought i would pitch in on what i’ve seen so far.

        svn revision 6066

        J 1 Reply Last reply Feb 23, 2017, 8:31 PM Reply Quote 0
        • J
          Junkhacker Developer @VincentJ
          last edited by Feb 23, 2017, 8:31 PM

          @VincentJ “/tmp/pigz1” is just the name of the fifo that the data is being piped into to be sent to compression. maybe we should rename it for purely aesthetic reasons, but that’s working as it should. was the previous image also Multiple Partition Single Disk (non re-sizeable)?

          signature:
          Junkhacker
          We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

          V 1 Reply Last reply Feb 24, 2017, 4:21 AM Reply Quote 1
          • V
            VincentJ Moderator @Junkhacker
            last edited by Feb 24, 2017, 4:21 AM

            @Junkhacker Yes it was. All of my images are.

            1 Reply Last reply Reply Quote 0
            • J
              JJ Fullmer Testers
              last edited by Mar 15, 2017, 8:11 PM

              So I just updated to RC 15 and made a new image at zstd level 9. I was using gzip level 5 before. The image is still currently downloading, estimating the full time to be about 12 minutes. With gzip the download was taking around 3 minutes. This is a 6th gen i5 client and the fogserver is on a vm with 8 vcpus (intel xeon something something) and 16 GB ram. There is also a storage node on a older server with a little less power, but still imaged around 2-3 minutes.
              The size of the image uncompressed is ~15 GB and it is 7.5 GB compressed with zstd. My old image was ~18 GB and with gzip level 5 it compressed to 6.7 GB.

              So what did I do wrong? Is the difference between 9 and 11 really that substantial? Should I re-upload that image? Also split vs non split? I saw the post on the topic of it being better for multiple images, but are there any other considerations in regard to speed?

              Have you tried the FogApi powershell module? It's pretty cool IMHO
              https://github.com/darksidemilk/FogApi
              https://fogapi.readthedocs.io/en/latest/
              https://www.powershellgallery.com/packages/FogApi
              https://forums.fogproject.org/topic/12026/powershell-api-module

              T 1 Reply Last reply Mar 15, 2017, 8:12 PM Reply Quote 0
              • T
                Tom Elliott @JJ Fullmer
                last edited by Mar 15, 2017, 8:12 PM

                @JJ-Fullmer ZSTD “mid” range is 11, compared to PIGZ “mid” range being 5-6.

                I set my images to 19 now and it’s pretty awesome.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                J 1 Reply Last reply Mar 15, 2017, 8:13 PM Reply Quote 1
                • J
                  JJ Fullmer Testers @Tom Elliott
                  last edited by Mar 15, 2017, 8:13 PM

                  @Tom-Elliott But at 19 are you sacrificing speed substantially?

                  Have you tried the FogApi powershell module? It's pretty cool IMHO
                  https://github.com/darksidemilk/FogApi
                  https://fogapi.readthedocs.io/en/latest/
                  https://www.powershellgallery.com/packages/FogApi
                  https://forums.fogproject.org/topic/12026/powershell-api-module

                  T 1 Reply Last reply Mar 15, 2017, 8:13 PM Reply Quote 0
                  • T
                    Tom Elliott @JJ Fullmer
                    last edited by Mar 15, 2017, 8:13 PM

                    @JJ-Fullmer Not at all.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    J 2 Replies Last reply Mar 15, 2017, 8:14 PM Reply Quote 1
                    • J
                      JJ Fullmer Testers @Tom Elliott
                      last edited by Mar 15, 2017, 8:14 PM

                      @Tom-Elliott I will give it a try then

                      Have you tried the FogApi powershell module? It's pretty cool IMHO
                      https://github.com/darksidemilk/FogApi
                      https://fogapi.readthedocs.io/en/latest/
                      https://www.powershellgallery.com/packages/FogApi
                      https://forums.fogproject.org/topic/12026/powershell-api-module

                      1 Reply Last reply Reply Quote 0
                      • J
                        JJ Fullmer Testers @Tom Elliott
                        last edited by Mar 15, 2017, 8:16 PM

                        @Tom-Elliott What about split vs not so split?

                        Have you tried the FogApi powershell module? It's pretty cool IMHO
                        https://github.com/darksidemilk/FogApi
                        https://fogapi.readthedocs.io/en/latest/
                        https://www.powershellgallery.com/packages/FogApi
                        https://forums.fogproject.org/topic/12026/powershell-api-module

                        T 1 Reply Last reply Mar 15, 2017, 8:18 PM Reply Quote 0
                        • T
                          Tom Elliott @JJ Fullmer
                          last edited by Mar 15, 2017, 8:18 PM

                          @JJ-Fullmer Either or, shouldn’t matter in terms of performance.

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          J 1 Reply Last reply Mar 15, 2017, 9:08 PM Reply Quote 1
                          • J
                            Junkhacker Developer @Tom Elliott
                            last edited by Mar 15, 2017, 9:08 PM

                            @Tom-Elliott i’m going to disagree with tom on the speed sacrifice with -19 setting. deploy speed, you won’t sacrifice much, and maybe even gain, but on capture it’s going to take quite a while longer. for me, i use -11.

                            signature:
                            Junkhacker
                            We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                            J 1 Reply Last reply Mar 15, 2017, 10:18 PM Reply Quote 1
                            • J
                              JJ Fullmer Testers @Junkhacker
                              last edited by Mar 15, 2017, 10:18 PM

                              @Junkhacker Well deploy is where speed is more important to me.
                              I gave 19 a go with the split and it actually had some weird error on the first deploy, I was in a meeting in my office so I didn’t really get to see the whole thing, but then it auto-retried the task and worked proper the second time. 2 minutes 18 seconds. The image size on the server seems to still be bigger than pigz was with 7.1 GB for a 15 GB image instead of 6.7 G for the old 18 GB image. But, don’t need to be too picky about it.

                              @Tom-Elliott Two minor issues I noticed. The first time (the slow deploy) I queued the image after doing a full host inventory via the pxe boot on the host. I had not specified any snapins and it randomly added like 10 of them. The second time I queued it from the gui and deleted the snapins and the problem didn’t repeat itself. However in both instances the drive didn’t resize itself.

                              Have you tried the FogApi powershell module? It's pretty cool IMHO
                              https://github.com/darksidemilk/FogApi
                              https://fogapi.readthedocs.io/en/latest/
                              https://www.powershellgallery.com/packages/FogApi
                              https://forums.fogproject.org/topic/12026/powershell-api-module

                              T 1 Reply Last reply Mar 15, 2017, 10:18 PM Reply Quote 1
                              • T
                                Tom Elliott @JJ Fullmer
                                last edited by Mar 15, 2017, 10:18 PM

                                @JJ-Fullmer What version of fog are you running?

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                1 Reply Last reply Reply Quote 0
                                • T
                                  Tom Elliott
                                  last edited by Mar 15, 2017, 11:17 PM

                                  @jj-fullmer and I had a remote session and the cause of the resize issue as described was found.

                                  The issue he was seeing was the resize allowed a partition to be sized larger than the boundary of the disk. This only occurred, as far as I could tell, in the case of 4k disks again, but I’ve added code to correct for this in ANY case of the chunk size passed in.

                                  Just thought I’d give an update to the “resize” issue as it was described and I’ve released RC-16 in an attempt to ensure this is corrected for.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  1 Reply Last reply Reply Quote 2
                                  • J
                                    JJ Fullmer Testers
                                    last edited by Mar 16, 2017, 8:45 PM

                                    Figured I should add, my speed issue was related to running PHPIPAM (a ip management tool) on my storage node server. It was slowing my image deploys down, not the zstd. It’s crazy fast at compression level 19 now.

                                    Have you tried the FogApi powershell module? It's pretty cool IMHO
                                    https://github.com/darksidemilk/FogApi
                                    https://fogapi.readthedocs.io/en/latest/
                                    https://www.powershellgallery.com/packages/FogApi
                                    https://forums.fogproject.org/topic/12026/powershell-api-module

                                    J 1 Reply Last reply Mar 21, 2017, 6:07 PM Reply Quote 2
                                    • J
                                      JJ Fullmer Testers @JJ Fullmer
                                      last edited by Mar 21, 2017, 6:07 PM

                                      @JJ-Fullmer Well it wasn’t actually phpipam that was slowing it down.
                                      I had some ssl logs running for a website running on that server. It was tracking access logs for every access of the ssl cert and site. Which was also adding up to over 400 GB of logs. I changed the apache configuration of the site to log much less. That solved the problem. So again, nothing wrong with zstd. Just wanted to have the right answer in case someone else happens to see a constant 5-8 Mbps transmit and receive on a storage node.

                                      Have you tried the FogApi powershell module? It's pretty cool IMHO
                                      https://github.com/darksidemilk/FogApi
                                      https://fogapi.readthedocs.io/en/latest/
                                      https://www.powershellgallery.com/packages/FogApi
                                      https://forums.fogproject.org/topic/12026/powershell-api-module

                                      1 Reply Last reply Reply Quote 1
                                      • 1
                                      • 2
                                      • 3
                                      • 4
                                      • 5
                                      • 4 / 5
                                      • First post
                                        Last post

                                      190

                                      Online

                                      12.0k

                                      Users

                                      17.3k

                                      Topics

                                      155.2k

                                      Posts
                                      Copyright © 2012-2024 FOG Project