Download/Upload Performance Issues Since 1.1.2



  • First off, once again, great work. I appreciate all the work that goes into this project.

    Now to get to business.

    I’ve noticed some significant performance issues since I upgraded from 1.0.1 to 1.1.2, and I can’t figure out where it’s coming from.

    Before the update, I was getting 5-7 GB/min for my downloads, and anywhere from 2-4 GB/min uploads. Now, I get ~1.39 GB/min downloads and ~750 MB/min to 1.3 GB/min uploads.

    Information:

    FOG Server-
    Ubuntu 12.04.4 64-bit
    Pentium Dual-Core E5500 2.8 GHz
    4 GB RAM
    Gigabit network connection
    Dedicated iSCSI gigabit connection to a HP Proliant DL385 with 6 15000 RPM drives in LVM running Ubuntu Server 12.04.4 (with dual processor, 9 GB RAM).

    This has been a great setup, but it took a significant hit as of 1.1.2.

    Throughout the setup, the network connection was bonded to two interfaces on each server in round-robin connection, both connected to the same switch, but I opted to try connecting the Proliant directly to the FOG server, but I haven’t seen any performance increase.

    Any suggestions on where else to look? Layer 1 is fine, and layer 2 seems to be working fine also.

    Thanks.

    Edit: Also, I’m doing unicast in all these setups. I haven’t had too great of performance in multicast to justify configuring it, and unicast works well for me.


  • Developer

    i’ve heard of problems with some cards, that they have a power setting (configurable from windows) that will drop to 100m in low power mode, and may not come back out of it and into 1000m when booting into linux. but when booted into windows, they come back up to full speed. if you disable that setting in windows, the problem goes away completely.



  • @ianabc, post: 31998, member: 24548 said:

    This is really puzzling, I’ll dig around and see if I have one of those cards available for testing. I think you already checked this, but could you confirm that in the fog debug ethtool reports 1000Mb as the negotiated speed and that you switch also agrees that this is the negotiated speed?

    I no longer have them here to test, but I can say for certain that that they auto-negotiated to gigabit. Even the switch was reading the speed as gigabit.


  • Testers

    This is really puzzling, I’ll dig around and see if I have one of those cards available for testing. I think you already checked this, but could you confirm that in the fog debug ethtool reports 1000Mb as the negotiated speed and that you switch also agrees that this is the negotiated speed?



  • @Tom Elliott, post: 31960, member: 7271 said:

    The kernel’s I’ve been building for quite some time have these drivers included already.

    Otherwise, they wouldn’t work at all.

    Maybe there’s a problem with the Linux driver? Windows doesn’t seem to have the 100 Mbps problem that I had. The issue only appeared in FOG.


  • Senior Developer

    The kernel’s I’ve been building for quite some time have these drivers included already.

    Otherwise, they wouldn’t work at all.



  • Finally got around to the testing this using your test, here’s my results:
    Thinkpad Edge E540they seem to have an issue with running at 100 Mbit even though it’s a gigabit interface[/U]. This correlates with my experience and the speeds that I was getting on those machines.

    Perhaps, then, the kernels need an update to include those drivers? I’m not sure, but at the very least I wanted to report my findings.

    PS- I think the hyperlink color should change, or at least add an underline like below. One word links seem to blend-in too well.

    Also, thanks again for all the help.


  • Testers

    Here is a sample NFS mount from a linux client - in principle it should all be GigE between the fog server and this machine, but they are quite a few hops apart and those switches and routers might be busy, could you try something similar for comparison?

    
      $ mkdir /fogtest
      $ mount -o vers=3,nolock IP.OF.YOUR.FOG:/images/dev /fogtest
      $ dd bs=1M count=1024 if=/dev/zero of=zeros.img conv=fdatasync
        1+0 records in
        1+0 records out
        1073741824 bytes (1.1 GB) copied 30.5422 s, 35.2 MB/s
     
      $ dd bs=1024M if=./zeros.img of=/dev/null iflag=direct
        1+0 records in
        1+0 records out
        1073741824 bytes (1.1 GB) copied 21.0712 s, 51.0 MB/s
     
      $ rm zeros.img
      $ umount /fogtest
      $ rmdir /fogtest
    
    

    I’m not too worried (or impressed :)) by these numbers. If you get similar results you know that the problem lies further up the stack.


  • Testers

    The iSCSI speed sounds plausible for a single GigE link, but the NFS does indeed seem slow.

    If there was a 100Mb link somewhere you would expect to see 7MB/s or so and as you said everything is claiming to be running GigE. 26MB/s is an odd result, especially if that is bi-directional. Can I ask how you are testing? dd over NFS can help but it can be a bit tricky because of the buffering and syncing.

    Did you try Tom’s suggestion of switching out the kernel for a test.

    Have a fun weekend!



  • Changed the compression and re-uploaded: no change.

    Finally was able to check the NFS transfer speed: I had 26 MB/sec transferring to and from one VM to the FOG server. Apparently my 75 MB/sec was my transfer speed to/from the FOG server to the iSCSI disk.

    So I’m left puzzled about what happened between now and then. I did some Ubuntu updates too.

    Guess I have something to be puzzled about over the weekend.


  • Senior Developer

    @ianabc, post: 31746, member: 24548 said:

    Is this is a raw copy over NFS? If not could you try a copy and report the speed? 75MB/s is in the right ballpark for gigabit over NFS.

    yeah, 75MB a second is pretty fast. Sorry I mis-interpretted it as 75MB/min.


  • Testers

    @braindead, post: 31723, member: 24282 said:

    My copies are going as they have before: ~75 MB/sec.

    Is this is a raw copy over NFS? If not could you try a copy and report the speed? 75MB/s is in the right ballpark for gigabit over NFS.


  • Senior Developer

    While the PIGZ_COMP may play into the speed, I doubt it’ll increase it that much. I did, however, change methods of networking within the kernel to hopefully fix some of the DHCP/BOOTP problems people where having. Maybe try a kernel from the 1.0.1 tag?



  • *The only change besides upgrading to 1.1.2.



  • I have restarted the switch already.

    The only change that I made recently was that I changed the compression of the image from 9 to 3, then I changed it to 1.

    I’m going to change that back and re-upload the image and see what that does.


  • Senior Developer

    I swear, nothing major changed from “functionality” of the init’s and I haven’t seen a speed increase or decrease on my side. It sounds to me like either the switch is failing or cables are faulty then. While I realize this probably isn’t the case, have you simply tried restarting the switch and see if it helps out?



  • Former speeds on 1.0.1*



  • Unfortunately, this all on our in-shop network, so no Apple devices are connected, and it’s all connected to one switch.

    I’m imaging Lenovo E540 laptops, and we had the former speeds with these same machines just a couple of days ago.

    @Tom Elliott, post: 31727, member: 7271 said:

    Do you have a apple devices in your network? Specifically running on the bonjour side of things?

    If you cut out the middle man as a test, do speeds improve? Here I mean take one of the “slow” clients and place them on the same switch as the FOG Server.


  • Senior Developer

    Do you have a apple devices in your network? Specifically running on the bonjour side of things?

    If you cut out the middle man as a test, do speeds improve? Here I mean take one of the “slow” clients and place them on the same switch as the FOG Server.



  • @Tom Elliott, post: 31724, member: 7271 said:

    Are you running multicast or unicast?

    Unicast, and those speeds are based on one machine at a time.


Log in to reply
 

447
Online

38716
Users

10544
Topics

99831
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.