• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

Download/Upload Performance Issues Since 1.1.2

Scheduled Pinned Locked Moved
Linux Problems
4
28
7.1k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • B
    braindead
    last edited by Jun 27, 2014, 5:36 PM

    First off, once again, great work. I appreciate all the work that goes into this project.

    Now to get to business.

    I’ve noticed some significant performance issues since I upgraded from 1.0.1 to 1.1.2, and I can’t figure out where it’s coming from.

    Before the update, I was getting 5-7 GB/min for my downloads, and anywhere from 2-4 GB/min uploads. Now, I get ~1.39 GB/min downloads and ~750 MB/min to 1.3 GB/min uploads.

    Information:

    FOG Server-
    Ubuntu 12.04.4 64-bit
    Pentium Dual-Core E5500 2.8 GHz
    4 GB RAM
    Gigabit network connection
    Dedicated iSCSI gigabit connection to a HP Proliant DL385 with 6 15000 RPM drives in LVM running Ubuntu Server 12.04.4 (with dual processor, 9 GB RAM).

    This has been a great setup, but it took a significant hit as of 1.1.2.

    Throughout the setup, the network connection was bonded to two interfaces on each server in round-robin connection, both connected to the same switch, but I opted to try connecting the Proliant directly to the FOG server, but I haven’t seen any performance increase.

    Any suggestions on where else to look? Layer 1 is fine, and layer 2 seems to be working fine also.

    Thanks.

    Edit: Also, I’m doing unicast in all these setups. I haven’t had too great of performance in multicast to justify configuring it, and unicast works well for me.

    1 Reply Last reply Reply Quote 0
    • I
      ianabc Testers
      last edited by Jun 27, 2014, 5:54 PM

      Has your iSCSI performance dropped? or is it just imaging? The second set of numbers you gave look typical of a single interface to me so my first guess would be your interface bonding has come undone during the update.

      1 Reply Last reply Reply Quote 0
      • B
        braindead
        last edited by Jun 27, 2014, 5:56 PM

        [quote=“ianabc, post: 31699, member: 24548”]Has your iSCSI performance dropped? or is it just imaging? The second set of numbers you gave look typical of a single interface to me so my first guess would be your interface bonding has come undone during the update.[/quote]

        No iSCSI performance drops. I changed to single interfaces in response to the performance drop, thinking the switch may be getting overworked, but nope, same performance. 😕

        1 Reply Last reply Reply Quote 0
        • I
          ianabc Testers
          last edited by Jun 27, 2014, 6:17 PM

          [quote=“braindead, post: 31694, member: 24282”]
          Before the update, I was getting 5-7 GB/min for my downloads, and anywhere from 2-4 GB/min uploads. Now, I get ~1.39 GB/min downloads and ~750 MB/min to 1.3 GB/min uploads.
          [/quote]

          Is 5-7 GB/min aggregate for deploying multiple images simultaneously? And are these the numbers reported by fog/partclone or have you also noticed that an actual image is taking longer?

          I’m on a single [S]gigabit[/S] fast ethernet (100Mb/s) link and partclone consistently tells me that it is running at about 1-2GB/min for a download at the moment - I haven’t noticed a significant change from other versions. The numbers that partclone reports will be for a the compressed image transfer so will be quite variable.

          If you wanted to dig a bit deeper you could take a look at the link negotiation on the switch and make sure it isn’t falling back to 100MB/s somewhere. The debug task in fog might also help you check the link speed and do some copies to check the numbers, for gigabit ethernet the max is somewhere around 117MB/s without jumbo frames, in the real world for NFS I’d expect somewhere around 90MB/s.

          EDIT ethtool is available in the fog debug task, “ethtool eth0” should tell you what link speed was negotiated without messing with your switch. Mine is reporting 100MB/s for some depressing reason 😞

          1 Reply Last reply Reply Quote 0
          • I
            ianabc Testers
            last edited by Jun 27, 2014, 6:51 PM

            OK, so I’ve put everything back on gigabit and my numbers are consistent with your “Before” values (7GB/min or so), I’m on 1.1.2 by they way.

            I strongly suspect 100BaseT is being negotiated somewhere between your client and fog server. ethtool on the fog server and in the debug task on your client should help you rule them out, then it is just the switch(es) that would need to be checked.

            1 Reply Last reply Reply Quote 0
            • B
              braindead
              last edited by Jun 27, 2014, 7:06 PM

              I’ll check this stuff out when I get back from lunch and get back to you.

              ethtool on the servers was reporting gigabit earlier today (in the terminals), but I’m going to check it out to be sure.

              1 Reply Last reply Reply Quote 0
              • B
                braindead
                last edited by Jun 27, 2014, 8:29 PM

                Well, bad news: ethtool is showing all interfaces to be at gigabit speeds, and the switch is showing gigabit speeds being negotiated.

                My copies are going as they have before: ~75 MB/sec.

                Also, my numbers were based on one machine imaging. With about 7 machines, it got down to ~1.2 GB/min.

                1 Reply Last reply Reply Quote 0
                • T
                  Tom Elliott
                  last edited by Jun 27, 2014, 8:30 PM

                  Are you running multicast or unicast?

                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                  1 Reply Last reply Reply Quote 0
                  • T
                    Tom Elliott
                    last edited by Jun 27, 2014, 8:31 PM

                    I know you edited a field to “answer” this, but are these speeds from multicast or unicast or both?

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    1 Reply Last reply Reply Quote 0
                    • B
                      braindead
                      last edited by Jun 27, 2014, 8:31 PM

                      [quote=“Tom Elliott, post: 31724, member: 7271”]Are you running multicast or unicast?[/quote]

                      Unicast, and those speeds are based on one machine at a time.

                      1 Reply Last reply Reply Quote 0
                      • T
                        Tom Elliott
                        last edited by Jun 27, 2014, 8:35 PM

                        Do you have a apple devices in your network? Specifically running on the bonjour side of things?

                        If you cut out the middle man as a test, do speeds improve? Here I mean take one of the “slow” clients and place them on the same switch as the FOG Server.

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        1 Reply Last reply Reply Quote 0
                        • B
                          braindead
                          last edited by Jun 27, 2014, 8:41 PM

                          Unfortunately, this all on our in-shop network, so no Apple devices are connected, and it’s all connected to one switch.

                          I’m imaging Lenovo E540 laptops, and we had the former speeds with these same machines just a couple of days ago.

                          [quote=“Tom Elliott, post: 31727, member: 7271”]Do you have a apple devices in your network? Specifically running on the bonjour side of things?

                          If you cut out the middle man as a test, do speeds improve? Here I mean take one of the “slow” clients and place them on the same switch as the FOG Server.[/quote]

                          1 Reply Last reply Reply Quote 0
                          • B
                            braindead
                            last edited by Jun 27, 2014, 8:42 PM

                            Former speeds on 1.0.1*

                            1 Reply Last reply Reply Quote 0
                            • T
                              Tom Elliott
                              last edited by Jun 27, 2014, 8:46 PM

                              I swear, nothing major changed from “functionality” of the init’s and I haven’t seen a speed increase or decrease on my side. It sounds to me like either the switch is failing or cables are faulty then. While I realize this probably isn’t the case, have you simply tried restarting the switch and see if it helps out?

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              1 Reply Last reply Reply Quote 0
                              • B
                                braindead
                                last edited by Jun 27, 2014, 8:56 PM

                                I have restarted the switch already.

                                The only change that I made recently was that I changed the compression of the image from 9 to 3, then I changed it to 1.

                                I’m going to change that back and re-upload the image and see what that does.

                                1 Reply Last reply Reply Quote 0
                                • B
                                  braindead
                                  last edited by Jun 27, 2014, 8:57 PM

                                  *The only change besides upgrading to 1.1.2.

                                  1 Reply Last reply Reply Quote 0
                                  • T
                                    Tom Elliott
                                    last edited by Jun 27, 2014, 9:05 PM

                                    While the PIGZ_COMP may play into the speed, I doubt it’ll increase it that much. I did, however, change methods of networking within the kernel to hopefully fix some of the DHCP/BOOTP problems people where having. Maybe try a kernel from the 1.0.1 tag?

                                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                    1 Reply Last reply Reply Quote 0
                                    • I
                                      ianabc Testers
                                      last edited by Jun 27, 2014, 9:30 PM

                                      [quote=“braindead, post: 31723, member: 24282”]
                                      My copies are going as they have before: ~75 MB/sec.
                                      [/quote]

                                      Is this is a raw copy over NFS? If not could you try a copy and report the speed? 75MB/s is in the right ballpark for gigabit over NFS.

                                      1 Reply Last reply Reply Quote 0
                                      • T
                                        Tom Elliott
                                        last edited by Jun 27, 2014, 9:50 PM

                                        [quote=“ianabc, post: 31746, member: 24548”]Is this is a raw copy over NFS? If not could you try a copy and report the speed? 75MB/s is in the right ballpark for gigabit over NFS.[/quote]

                                        yeah, 75MB a second is pretty fast. Sorry I mis-interpretted it as 75MB/min.

                                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          braindead
                                          last edited by Jun 27, 2014, 11:10 PM

                                          Changed the compression and re-uploaded: no change.

                                          Finally was able to check the NFS transfer speed: I had 26 MB/sec transferring to and from one VM to the FOG server. Apparently my 75 MB/sec was my transfer speed to/from the FOG server to the iSCSI disk.

                                          So I’m left puzzled about what happened between now and then. I did some Ubuntu updates too.

                                          Guess I have something to be puzzled about over the weekend.

                                          1 Reply Last reply Reply Quote 0
                                          • 1
                                          • 2
                                          • 1 / 2
                                          1 / 2
                                          • First post
                                            4/28
                                            Last post

                                          155

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project