• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Imaging computers at 2.6Gbps

    Scheduled Pinned Locked Moved
    General
    8
    40
    8.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • JunkhackerJ
      Junkhacker Developer @george1421
      last edited by

      @george1421 by using zstd compression 11 i have managed to make my images ~26% smaller and deploy ~36% faster. of course, that’s with a normal image. one full of video may not do so well. mine starts at about 18-20GB/min, but that settles down to the speed in the picture after it gets about 5% done.

      signature:
      Junkhacker
      We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

      O 1 Reply Last reply Reply Quote 0
      • O
        Obi-Jon @Junkhacker
        last edited by

        @Junkhacker Are you using SSDs on your server (and client I would assume)? Wow, smaller images and still faster, that’s pretty sweet.

        1 Reply Last reply Reply Quote 0
        • JunkhackerJ
          Junkhacker Developer @Obi-Jon
          last edited by

          @Obi-Jon it’s important to note that gzip (the default) only has legitimate values of 0-9. zstd has legitimate values of 0-19 (normal) 20-22 (ultra). i don’t have a computer that can use ultra settings without running out of ram and crashing, and i don’t think the extra gains from those settings would be worth it if i did.

          signature:
          Junkhacker
          We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

          JunkhackerJ 1 Reply Last reply Reply Quote 0
          • JunkhackerJ
            Junkhacker Developer @Junkhacker
            last edited by

            @Junkhacker no SSD on the server, it’s running in a vm with a raid for storage

            signature:
            Junkhacker
            We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

            O 1 Reply Last reply Reply Quote 0
            • O
              Obi-Jon @Junkhacker
              last edited by

              @Junkhacker Yeah, I was curious if SSD really helps much at all with straight throughput. I figured SSD might win out when pushing multiple images at once, which is what I plan to do. Should be interesting to see how it goes on several dozen simultaneous unicasts.

              Good to know about the gzip values vs zstd. I’m testing a plain Win10 image now with zstd and your settings to see how it compares to gzip. Will test with the big image when I have more time.

              JunkhackerJ 1 Reply Last reply Reply Quote 0
              • JunkhackerJ
                Junkhacker Developer @Obi-Jon
                last edited by

                @Obi-Jon you may want to use zstd -19 compression. that takes about 3x longer on upload for me, but it shaves another ~5% off the size of my images. it takes about the same time when deploying to a single machine, but that could make a big difference in the volume of network traffic with multiple unicasts

                signature:
                Junkhacker
                We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                O 1 Reply Last reply Reply Quote 0
                • O
                  Obi-Jon @Junkhacker
                  last edited by

                  @Junkhacker Testing zstd -11 vs -19 now. Definitely slower uploading 19, but that wouldn’t matter much to me since I can start it and walk away. Does compression take place on the client or server? I know downloads decompress at the client obviously, but assuming compression also takes place on the client there would be little need for a fast CPU on the server. I’m only running an i3 on my server, used the money for memory, drive and nic instead.

                  Tom ElliottT george1421G 2 Replies Last reply Reply Quote 0
                  • Tom ElliottT
                    Tom Elliott @Obi-Jon
                    last edited by

                    @Obi-Jon compression or decompression is always handled on the client system.

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    1 Reply Last reply Reply Quote 0
                    • george1421G
                      george1421 Moderator @Obi-Jon
                      last edited by george1421

                      @Obi-Jon All heavy lifting (computing) is done on the client. The server is only used to move the files between the network and storage and overall process management.

                      I’ve run fog on a raspberry Pi2 with pretty good results with a single unicast stream.

                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                      O 1 Reply Last reply Reply Quote 0
                      • O
                        Obi-Jon @george1421
                        last edited by

                        @george1421 Good to know.

                        Raspberry Pi2 FOG would have been sweet for carrying into remote 56k-connected offices where I used to work back in the early 2000s.

                        george1421G 1 Reply Last reply Reply Quote 0
                        • george1421G
                          george1421 Moderator @Obi-Jon
                          last edited by george1421

                          @Obi-Jon We use it for a mobile deployment server and a bit of a novelty. For small offices we typically use an i3 Intel NUC with an onboad SSD drive. Its pretty small, easy on power and has enough horse power to support multiple deployment streams.

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                          1 Reply Last reply Reply Quote 0
                          • O
                            Obi-Jon
                            last edited by Obi-Jon

                            OK, test results are in, and wow, zstd is fast, at least on my newest i3 systems with a vanilla Win10 installation (haven’t yet tested it with the original 157GB image I posted earlier). So, this isn’t really apples to apples when comparing with my earlier results, but solid comparison between zstd compression levels.

                            Vanilla Win10 SSD/6th gen i3/32GB ram system:
                            Image: 9,180MB uncompressed (converted from 8.55GiB as shown in FOG to MB)
                            zstd -11: 3,390MB compressed, avg upload 3.66GB/min, download time 27 seconds, peak 18.23GB/min
                            zstd -19: 3,068MB compressed, avg upload 725MB/min, download time 25 seconds, peak 19.75GB/min

                            So it looks like an speed improvement of about 7% and a space improvement of 9%, tradeoff being 400% increase in upload time. Worth it for some, not for others. Totally worth it for me to save server space and make deployments as quick as possible to reduce user downtime. I suspect these increases will vary depending on client hardware specs.

                            HOWEVER, I did see an “error 39 read premature end” at the very end of the download process for zstd -19 right before it rebooted. I didn’t notice if there was an error when uploading, but the error did cause FOG to repeat the image process until I killed it. However, Windows 10 booted fine and I don’t see any problems. I re-uploaded the image and compared disk usage and the post-error image is 10MB smaller, so I wouldn’t trust the image. This error may have been a fluke, will probably settle in the zfs -15 range if -19 continues to generate errors.

                            And for fun…
                            0_1491517144099_Speedtest 9180MB image zstd-19.JPG

                            JunkhackerJ Tom ElliottT B 3 Replies Last reply Reply Quote 3
                            • JunkhackerJ
                              Junkhacker Developer @Obi-Jon
                              last edited by

                              @Obi-Jon the error is most likely completely unrelated to the compression level. if you can capture more information, we’ll see if we can find the cause.

                              signature:
                              Junkhacker
                              We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                              O 1 Reply Last reply Reply Quote 0
                              • O
                                Obi-Jon @Junkhacker
                                last edited by Obi-Jon

                                @Junkhacker “locate partclone.log” does not find a log file. Thinking a log was not generated, or am I looking for the wrong log file? I tried “locate *.log” but didn’t see anything promising:

                                /opt/fog/log/fogimagesize.log
                                /opt/fog/log/fogreplicator.log
                                /opt/fog/log/fogscheduler.log
                                /opt/fog/log/fogsnapinhash.log
                                /opt/fog/log/fogsnapinrep.log
                                /opt/fog/log/groupmanager.log
                                /opt/fog/log/multicast.log
                                /opt/fog/log/pinghost.log
                                /opt/fog/log/servicemaster.log
                                /root/fogproject/bin/error_logs/fog_error_1.3.5.log
                                /root/fogproject/bin/error_logs/foginstall.log
                                /var/log/alternatives.log
                                /var/log/auth.log
                                /var/log/bootstrap.log
                                /var/log/dpkg.log
                                /var/log/kern.log
                                /var/log/php7.1-fpm.log
                                /var/log/apache2/access.log
                                /var/log/apache2/error.log
                                /var/log/apache2/other_vhosts_access.log
                                /var/log/apt/history.log
                                /var/log/apt/term.log
                                /var/log/mysql/error.log
                                /var/log/unattended-upgrades/unattended-upgrades-shutdown.log

                                0_1491518979404_Speedtest error.JPG

                                Tom ElliottT 1 Reply Last reply Reply Quote 0
                                • Tom ElliottT
                                  Tom Elliott @Obi-Jon
                                  last edited by

                                  @Obi-Jon Partclone.log is related to the client.

                                  Anything presented on the client in regards to FOS (Inits) will be in respect to the client.

                                  You could add isdebug=yes to that host’s Kernel Arguments field. This will drop the host into a terminal prompt where we can step through and obtain more information.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  O 1 Reply Last reply Reply Quote 0
                                  • O
                                    Obi-Jon @Tom Elliott
                                    last edited by Obi-Jon

                                    @Tom-Elliott Ah, I borked the log then when I uploaded the image and redownloaded after the error to compare image file sizes. No errors the second time since the images matched. If I see that error again I’ll revisit this. Thanks!

                                    1 Reply Last reply Reply Quote 0
                                    • Tom ElliottT
                                      Tom Elliott @Obi-Jon
                                      last edited by

                                      @Obi-Jon I feel I must add, once again.

                                      “Holy Shit”

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                      O 1 Reply Last reply Reply Quote 2
                                      • O
                                        Obi-Jon @Tom Elliott
                                        last edited by

                                        @Tom-Elliott Lol, my sentiments as well, been pumped all day. Can’t wait to try this over multiple unicasts simultaneously.

                                        1 Reply Last reply Reply Quote 1
                                        • B
                                          Bob Henderson @Obi-Jon
                                          last edited by

                                          @Obi-Jon That’s great to see. We see very similar speeds with ours, though it’s just a VM on ZFS based storage with 4g of ram and 2 cores. I think the key is making sure you’ve got solid clients, and ssds on the client side. And Intel nics.

                                          Some of our broadcom equipped thinkpads max out at around 7, the intel Dells hit the 10,11 mark.

                                          This is all on gzip. Haven’t made a new image since the new compression.

                                          O 1 Reply Last reply Reply Quote 0
                                          • O
                                            Obi-Jon @Bob Henderson
                                            last edited by

                                            @Bob-Henderson Good point on the Intel nics, that’s what I’m using as well. My new clients are based on H110T chipset, which has both Intel and Realtek nics. I made sure to only enable PXE on the Intel nic and disabled the Realtek nic entirely. Now to keep my users from plugging into the wrong jack and generating more help desk tickets. I’m actually contemplating gluing a punched down RJ45 plug into the Realtek ports, lol.

                                            B 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post

                                            166

                                            Online

                                            12.0k

                                            Users

                                            17.3k

                                            Topics

                                            155.2k

                                            Posts
                                            Copyright © 2012-2024 FOG Project