• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

Imaging computers at 2.6Gbps

Scheduled Pinned Locked Moved
General
8
40
8.8k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • O
    Obi-Jon
    last edited by Obi-Jon Apr 6, 2017, 4:27 PM Apr 6, 2017, 3:37 PM

    Yes, FOG can do it. I just built a new server with 1TB SSD, 32GB memory and 10Gbps fiber network adapter for well under $2,000. It’s a 1U Supermicro barebones system with an i3-based processor running Ubuntu Server 16.10. All my computers are SSD-based finally, and following are the results of my first test download of an image (157GB image across two separate switches with 1GbE port at the client).

    Entire image completed in under 15 minutes, never dropped below 9.5GB/min. Top speed was about 10.9GB/min, or 1.5Gbps [edit: up to 19.75GB/min or 2.6Gbps thanks to zstd, see more recent post below). This was only possible over a 1Gbps link due to the compression of the image. I ran this test in the middle of the day while about 400 people were on the network. I didn’t look at the network port utilization but it had to be close to 100%. Client computer is an i5-3570K (3.4Ghz) with 256GB MSATA SSD and 8GB memory. The server’s 10Gbps link isn’t all that necessary for single deployments like this but for deploying to multiple classrooms at once it should allow me to image dozens of computers at once in unicast mode (most of our edge ports are still 100Mbps).

    No RAID on the server, I figure if it croaks I can rebuild. One backup of the server and database and backups of my images are enough for me to sleep soundly at night, which I will be able to do more now that I can image this quickly! Anyone else seeing these kind of speeds? I continue to be impressed with FOG.

    0_1491492847436_Speedtest.jpg

    1 Reply Last reply Reply Quote 5
    • T
      Tom Elliott
      last edited by Apr 6, 2017, 4:20 PM

      I just have to say,

      HOLY SHIT!

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

      Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

      Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

      G 1 Reply Last reply Apr 6, 2017, 5:29 PM Reply Quote 2
      • K
        kai
        last edited by Apr 6, 2017, 4:56 PM

        I’m just trying hard in not being jealous…
        🙂

        1 Reply Last reply Reply Quote 0
        • G
          george1421 Moderator @Tom Elliott
          last edited by Apr 6, 2017, 5:29 PM

          @Tom-Elliott said in Imaging computers at 1.5Gbps:

          I just have to say,

          HOLY SHIT!

          Ditto…

          Of course me being a bit OCD, I have to know was this transfer rate done with FOG standard compression or the new (as of 1.3.4) zstd file compression? To get the best speeds from the zstd compression you need to capture and deploy using the zstd drivers. If this was done with the legacy data compressor, what was your compression index number?

          The reason why I asked the 20 questions is that what you have is might impressive. If others want to duplicate your test, it would be good to know the conditions.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

          O 1 Reply Last reply Apr 6, 2017, 8:54 PM Reply Quote 3
          • J
            Junkhacker Developer
            last edited by Junkhacker Apr 6, 2017, 11:44 AM Apr 6, 2017, 5:43 PM

            since we’re sharing…
            0_1491500624344_scaled.jpg
            i5 SSD client
            zstd -11 compression

            signature:
            Junkhacker
            We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

            G 1 Reply Last reply Apr 6, 2017, 5:46 PM Reply Quote 3
            • G
              george1421 Moderator @Junkhacker
              last edited by george1421 Apr 6, 2017, 12:19 PM Apr 6, 2017, 5:46 PM

              @Junkhacker show off… 😕

              < really means George is envious of your tech >

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

              1 Reply Last reply Reply Quote 2
              • O
                Obi-Jon @george1421
                last edited by Apr 6, 2017, 8:54 PM

                @george1421 Standard FOG partclone gzip here. I may have to play with zstd. As for compression index, the image that I tested above is 153.6GB (confirmed in Windows as well) and on the FOG server is using 94GB, so I believe that would be 94/153.6 = 61.2% compression index (if that is the correct method for calculating compression index). Or perhaps it would be 100-61.2 = 38.8% compression index? At any rate, I’m surprised it compressed as well as it did because most of this user’s hard disk is comprised of compressed video (radio/television teacher).

                With a newer client PC (M.2 drive) I attained 13GB/min, still short of Junkhacker’s 15.34GB/min. Wow. I will have to try it again after hours and see how much network traffic affected my earlier tests.

                G 1 Reply Last reply Apr 6, 2017, 8:57 PM Reply Quote 0
                • G
                  george1421 Moderator @Obi-Jon
                  last edited by george1421 Apr 6, 2017, 2:59 PM Apr 6, 2017, 8:57 PM

                  @Obi-Jon said in Imaging computers at 1.5Gbps:

                  Junkhacker’s 15.34GB/min

                  I’m pretty sure Junkhacker is now using zstd for capture and deployment. So that may explain the speed differences.

                  FWIW: The compression ratio I was talking about was the slider position on the image definition from the FOG management gui.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                  O J 2 Replies Last reply Apr 6, 2017, 9:23 PM Reply Quote 0
                  • O
                    Obi-Jon @george1421
                    last edited by Apr 6, 2017, 9:23 PM

                    @george1421 Ah, I didn’t even notice that when I set up the test image. It is set to 6 out of 22 (the default I presume since I didn’t change it).

                    Since I didn’t bring any of my old images over from the old FOG server I may try zstd to see how it compares.

                    J 1 Reply Last reply Apr 6, 2017, 9:25 PM Reply Quote 0
                    • J
                      Junkhacker Developer @george1421
                      last edited by Apr 6, 2017, 9:23 PM

                      @george1421 by using zstd compression 11 i have managed to make my images ~26% smaller and deploy ~36% faster. of course, that’s with a normal image. one full of video may not do so well. mine starts at about 18-20GB/min, but that settles down to the speed in the picture after it gets about 5% done.

                      signature:
                      Junkhacker
                      We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                      O 1 Reply Last reply Apr 6, 2017, 9:25 PM Reply Quote 0
                      • O
                        Obi-Jon @Junkhacker
                        last edited by Apr 6, 2017, 9:25 PM

                        @Junkhacker Are you using SSDs on your server (and client I would assume)? Wow, smaller images and still faster, that’s pretty sweet.

                        1 Reply Last reply Reply Quote 0
                        • J
                          Junkhacker Developer @Obi-Jon
                          last edited by Apr 6, 2017, 9:25 PM

                          @Obi-Jon it’s important to note that gzip (the default) only has legitimate values of 0-9. zstd has legitimate values of 0-19 (normal) 20-22 (ultra). i don’t have a computer that can use ultra settings without running out of ram and crashing, and i don’t think the extra gains from those settings would be worth it if i did.

                          signature:
                          Junkhacker
                          We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                          J 1 Reply Last reply Apr 6, 2017, 9:26 PM Reply Quote 0
                          • J
                            Junkhacker Developer @Junkhacker
                            last edited by Apr 6, 2017, 9:26 PM

                            @Junkhacker no SSD on the server, it’s running in a vm with a raid for storage

                            signature:
                            Junkhacker
                            We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                            O 1 Reply Last reply Apr 6, 2017, 9:29 PM Reply Quote 0
                            • O
                              Obi-Jon @Junkhacker
                              last edited by Apr 6, 2017, 9:29 PM

                              @Junkhacker Yeah, I was curious if SSD really helps much at all with straight throughput. I figured SSD might win out when pushing multiple images at once, which is what I plan to do. Should be interesting to see how it goes on several dozen simultaneous unicasts.

                              Good to know about the gzip values vs zstd. I’m testing a plain Win10 image now with zstd and your settings to see how it compares to gzip. Will test with the big image when I have more time.

                              J 1 Reply Last reply Apr 6, 2017, 9:33 PM Reply Quote 0
                              • J
                                Junkhacker Developer @Obi-Jon
                                last edited by Apr 6, 2017, 9:33 PM

                                @Obi-Jon you may want to use zstd -19 compression. that takes about 3x longer on upload for me, but it shaves another ~5% off the size of my images. it takes about the same time when deploying to a single machine, but that could make a big difference in the volume of network traffic with multiple unicasts

                                signature:
                                Junkhacker
                                We are here to help you. If you are unresponsive to our questions, don't expect us to be responsive to yours.

                                O 1 Reply Last reply Apr 6, 2017, 9:41 PM Reply Quote 0
                                • O
                                  Obi-Jon @Junkhacker
                                  last edited by Apr 6, 2017, 9:41 PM

                                  @Junkhacker Testing zstd -11 vs -19 now. Definitely slower uploading 19, but that wouldn’t matter much to me since I can start it and walk away. Does compression take place on the client or server? I know downloads decompress at the client obviously, but assuming compression also takes place on the client there would be little need for a fast CPU on the server. I’m only running an i3 on my server, used the money for memory, drive and nic instead.

                                  T G 2 Replies Last reply Apr 6, 2017, 9:44 PM Reply Quote 0
                                  • T
                                    Tom Elliott @Obi-Jon
                                    last edited by Apr 6, 2017, 9:44 PM

                                    @Obi-Jon compression or decompression is always handled on the client system.

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                    1 Reply Last reply Reply Quote 0
                                    • G
                                      george1421 Moderator @Obi-Jon
                                      last edited by george1421 Apr 6, 2017, 3:45 PM Apr 6, 2017, 9:45 PM

                                      @Obi-Jon All heavy lifting (computing) is done on the client. The server is only used to move the files between the network and storage and overall process management.

                                      I’ve run fog on a raspberry Pi2 with pretty good results with a single unicast stream.

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                      O 1 Reply Last reply Apr 6, 2017, 10:03 PM Reply Quote 0
                                      • O
                                        Obi-Jon @george1421
                                        last edited by Apr 6, 2017, 10:03 PM

                                        @george1421 Good to know.

                                        Raspberry Pi2 FOG would have been sweet for carrying into remote 56k-connected offices where I used to work back in the early 2000s.

                                        G 1 Reply Last reply Apr 6, 2017, 10:06 PM Reply Quote 0
                                        • G
                                          george1421 Moderator @Obi-Jon
                                          last edited by george1421 Apr 6, 2017, 4:06 PM Apr 6, 2017, 10:06 PM

                                          @Obi-Jon We use it for a mobile deployment server and a bit of a novelty. For small offices we typically use an i3 Intel NUC with an onboad SSD drive. Its pretty small, easy on power and has enough horse power to support multiple deployment streams.

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                          1 Reply Last reply Reply Quote 0
                                          • 1
                                          • 2
                                          • 1 / 2
                                          1 / 2
                                          • First post
                                            20/40
                                            Last post

                                          152

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project