• Recent
    • Unsolved
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Imaging woes - unicast

    Scheduled Pinned Locked Moved
    FOG Problems
    3
    9
    925
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mesaman0182
      last edited by mesaman0182

      So I’m getting frustrated with imaging these new laptops. At first - I could image about 7-to-8 laptops at a time. Randomly I would have 1 or 2 stop in its tracks - the data transfer rate slowly decreases and there’s no change to the progress of the blocks being written. I am now imaging about 5 at a time and still seem to be encountering the same problem with a random-single machine having the same symptoms.

      Here’s my current setup:

      Fog Server is a VM using VirtualBox from a Dell Server PowerEdge 440 running on Ubuntu CLI OS 16.04.7 (yes I know its an older version; when I got hired I had to get an imaging server up ASAP to start imaging so I followed a YouTube video that was an easy setup just to get something going.)

      Hardware resources - 2 CPUs, 4GB RAM, Bridged Network Adapter - Broadcom NetXtreme Gigabit Ethernet

      Laptops don’t have a NIC so I am using USB-to-NIC adapters. Laptops are Dell Latitude 3190s.

      Imaging through a Cisco 2960-X Series switch - server and client machines are all plugged into the same switch. Machines are only utilizing the network to get DHCP from our domain controller.

      I did try multicasting when I first setup my imaging lab environment but the speed was abysmal. I’m not sure of the config of the switch as its part of a switch stack and haven’t yet explored the configuration of that switch.

      Any advice would be wonderful.

      Tom ElliottT george1421G 2 Replies Last reply Reply Quote 0
      • Tom ElliottT
        Tom Elliott @mesaman0182
        last edited by

        @mesaman0182 The issue with Unicast is that it’s a shared resource.

        So in Multicast, a single file is just opened and sent across the network. Speeds are usually abysmal because the traffic across the switch is just flodded.

        In unicast, however, it’s X times opened. Where X = the number of machines requesting the same file.

        So let’s say you have 1 machine requesting 1 image. (All machines after this point will request the exact same image/file.)

        Only one stream needs to be made.

        Now that’s add 2. Well the file is now being requested 2 times and being sent across the network. So that effectively splits the amount of “bandwidth” in half. Let’s add a 3rd machine to the mix, we are now only getting on a 1GB connection, each machine is getting 1/3 of a Gigabit.

        So 1000 / 2 = 500

        1000 / 3 = 333.33

        1000 / 4 = 250

        and so on.

        Now all of this assumes you’re getting 100% throughput on all scenarios. This is highly unlikely but hopefully illustrates the exact issue you’re seeing.

        Now FOG imaging isn’t 100% perfect in its throughput estimations either. The estimations we see are based on Network throughput + CPU Processing to decompress the image + HDD time to write the data.

        In a 100% perfect situation you will get 7.25 Gigabytes per minute on a 1GB connection.

        Let’s try to make this easier to understand without all the technical jargon.

        Multicast =

        You have one copy of a document and are presenting it to a room of 5 people. 3 people might not get all the information but that doesn’t matter.

        Vs.

        Unicast =
        You have one copy of a document and are presenting it to a room of 5 people, but each person that connects has to start from the beginning and anybody currently accessing it is slowed down by the number of people accessing it at that time. Also, anybody who has a single error must be resent the failed point.

        Hopefully this helps.

        It may make more sense to actually image 1 or 2 machines at a time, rather than 8 - 10 at a time. Mainly because it will just go a lot faster. Or figure out why multicast wasn’t working and use it to image large groups at once.

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        M 1 Reply Last reply Reply Quote 0
        • george1421G
          george1421 Moderator @mesaman0182
          last edited by george1421

          @mesaman0182 Well… You can saturate a 1GbE link imaging 3 contemporary computers at at time. On top of that you are running a type 2 hypervisor which also impacts the performance and throughput.

          For a single image what is your transfer rate according to Partclone?

          On a well managed 1GbE network you should be able to transfer about 6.1GB/min for a single unicast.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

          M 1 Reply Last reply Reply Quote 1
          • M
            mesaman0182 @george1421
            last edited by mesaman0182

            @george1421 It varies but I am imaging anywhere from 1-2.75GB - that’s usually with about 4 machines going.

            As far as the VM goes - we have recently retired some older servers - one which is a 2008 server that I could repurpose for imaging; would just have to put Ubuntu on it then install Fog. I am currently in the process of installing a Synology NAS appliance for backup and storage which will have an SFP card in it that will do 10GB througput. Once that project is complete I am considering setting up a new fog server with a storage node (the NAS). May just have to tolerate things as is for now and do what Tom is suggesting.

            Thank you for your quick response.

            1 Reply Last reply Reply Quote 0
            • M
              mesaman0182 @Tom Elliott
              last edited by

              @tom-elliott “So in Multicast, a single file is just opened and sent across the network. Speeds are usually abysmal because the traffic across the switch is just flodded.” - This sounds like a description of broadcasting which goes out to all network devices.

              My understanding of multicast is traffic is only sent out to a “select” group of devices; those requesting the data. Why then would that flood the switch? Sorry, its been 6 years since I went to school for Cisco; I need to freshen up and re-educate myself.

              george1421G Tom ElliottT 2 Replies Last reply Reply Quote 0
              • george1421G
                george1421 Moderator @mesaman0182
                last edited by george1421

                @mesaman0182 said in Imaging woes - unicast:

                Why then would that flood the switch?

                It will flood on non enterprise switches because the processors are usually under powered and can’t support more of the advanced features needed for multicasting, so it just sends the data out to all ports. That works but not very helpful. There is also consideration for sparse-mode vs dense-mode for multicasting. Sparse mode is desired if all of your multicast devices (receivers) are not on the same switch. You may also need to enable igmp-snooping on the vlan where your multicast devices are connected. A properly configured network can multicast without killing the entire network, because when the subscribers announce they are there the igmp snooping will register that device with the switch for multicasting.

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                M 1 Reply Last reply Reply Quote 0
                • Tom ElliottT
                  Tom Elliott @mesaman0182
                  last edited by

                  @mesaman0182 Multicast is “Broadcasting across a network” using UDP to transfer the information. The one’s “requesting” the data will say, yep that’s for me, but it literally does broadcast against the whole network.

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  1 Reply Last reply Reply Quote 0
                  • M
                    mesaman0182 @george1421
                    last edited by mesaman0182

                    @george1421 - After looking at the running config on my router; “ip pim sparse-dense-mode” is enabled on the data VLAN. I’ve also looked at the switch stack and IGMP Snooping is configured (multicast devices are all plugged into a switch that’s part of a stack).

                    1 Reply Last reply Reply Quote 0
                    • M
                      mesaman0182
                      last edited by mesaman0182

                      UPDATE: After doing some investigation of the network device(s) configuration; I discovered the multicast looks to be configured properly on the router and switch stack.

                      For curiosity sake, I have attempted a multicast session for 4 laptops - my transfer rate is approximately 1.85GB/per min. This is exciting. I am going to gradually increase the load from there. If I can get away with 8 devices at a time that would be ideal.

                      Its strange because I know for a fact upon my first attempt a month ago, multicasting was terrible; it was virtually stationary.

                      1 Reply Last reply Reply Quote 0
                      • 1 / 1
                      • First post
                        Last post

                      156

                      Online

                      12.0k

                      Users

                      17.3k

                      Topics

                      155.2k

                      Posts
                      Copyright © 2012-2024 FOG Project