How-to: Network bonding & RAID arrays

  • Developer

    [I]Please note: This is an advanced topic and requires basic Linux knowledge. These examples only provide information on how my system was configured.[/I]

    Using the following setup i can unicast 15-20 machines at full speed (per node, 40 in total with 2 nodes). Unicast speeds range from 5.5gb/minute (~90mb/sec) to 3gb/minute (~50mb/sec). Normally it is much faster to unicast over multicast due to multicast’s fiddly deployments.

    [SIZE=4][B]FOG VM Controller[/B] - FOG-1[/SIZE]
    []Stores no images, only used for a controller
    []5gb HDD
    ]1gb RAM
    [SIZE=4][B]FOG Storage Nodes[/B] - FOG-2 & FOG-3[/SIZE]
    ]These machines are custom white boxes, they were left over workstations which were turned into FOG storage nodes. Nothing fancy here.
    []CPU: E8400 Core2Duo
    ]Motherboard: Intel DG35EC
    []RAM: DDR2 4gb - overkill really
    ]OS Drive: 80gb Western Digital Green - Image storage is on the RAID array below
    ]2x Intel 82574L PCIe 1gb network cards - bonded
    []Uses package ‘ifenslave’ (apt-get install ifenslave)
    ]Bond Mode 0 - mode 4 (802.3ad) works [I]slightly[/I] better, but requires switch support
    []eth1 & 2
    ]eth0 still has a cable in case something happens to the bond, the server can still be accessed remotely

    [code]fog-2:~# cat /etc/network/interfaces

    This file describes the network interfaces available on your system

    and how to activate them. For more information, see interfaces(5).

    The loopback network interface

    auto lo
    iface lo inet loopback

    The primary network interface

    iface eth0 inet dhcp

    bonded network connections

    allow-hotplug bond0
    auto bond0
    iface bond0 inet static
    slaves eth1 eth2
    bond-mode 0
    bond-miimon 100
    pre-up ifdown eth0
    post-down ifup eth0
    [*]module setup - not sure if this is still required with the new ifenslave method

    [code]fog-2:~# cat /etc/modprobe.d/arch-aliases

    network bonding

    alias bond0 bonding
    options bonding mode=0 miimon=100
    [*]ifconfig (lo removed)

    [code]fog-2:~# ifconfig
    bond0 Link encap:Ethernet HWaddr 00:1b:21:53:c0:e5
    inet addr: Bcast: Mask:
    inet6 addr: fe80::21b:21ff:fe53:c0e5/64 Scope:Link
    RX packets:11906 errors:0 dropped:0 overruns:0 frame:0
    TX packets:6105 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:2167156 (2.0 MiB) TX bytes:1736174 (1.6 MiB)

    eth1 Link encap:Ethernet HWaddr 00:1b:21:53:c0:e5
    RX packets:10145 errors:0 dropped:0 overruns:0 frame:0
    TX packets:3053 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:1912960 (1.8 MiB) TX bytes:869040 (848.6 KiB)

    eth2 Link encap:Ethernet HWaddr 00:1b:21:53:c0:e5
    RX packets:1761 errors:0 dropped:0 overruns:0 frame:0
    TX packets:3052 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:254196 (248.2 KiB) TX bytes:867134 (846.8 KiB)
    ]2x 2TB Western Digital Black
    []Non raid edition drives, just the standard blacks
    ]RAID 0
    []Software RAID using MDADM
    ]Create file systems

    [code]mke2fs -t ext4 -b 4096 -m 0 -i 65536 -E stride=64,stripe-width=128 /dev/md0[/code]

    Create a partition on md0 as EXT4, 4k block sizes, no root reserved space, 65k inode block sizes

    The images will be 1 big massive block of data. Maximizing block and inode sizes reduces overhead at the expense of a little wasted space (if the file partly uses a block).

    stride = This is the number of blocks read or written to disk before moving to next disk
    stripe_width = This is typically be stride-size * N, where N is the number of data disks in the RAID

    If you are unsure of how to calculate stride and stripe_width, there is a handy calculator here: [url][/url]
    [*]Create MDADM Array

    chunk size = 256kb # this could be 512kb, but i have seen no performance increase on this hardware set

    A larger chunk size results in less overhead (faster performance) at the expense of wasted disk. The amount of waste will be minimal was we are using EXT4 with preallocate enabled. EXT preallocate has no overhead, where EXT3 must manually zero the files. Get EXT4!!

    [code]mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=2 --chunk=256 /dev/sdb1 /dev/sdc1[/code]

    [*]MDADM config


    Generated with:

    fog-2:~# mdadm --detail --scan >> /etc/mdadm/mdadm.conf

    fog-2:~# cat /etc/mdadm/mdadm.conf


    Please refer to mdadm.conf(5) for information about this file.

    by default, scan all partitions (/proc/partitions) for MD superblocks.

    alternatively, specify devices to scan, using wildcards if desired.

    DEVICE partitions

    auto-create devices with Debian standard permissions

    CREATE owner=root group=disk mode=0660 auto=yes

    automatically tag new arrays as belonging to the local system

    HOMEHOST <system>

    instruct the monitoring daemon where to send mail alerts

    MAILADDR root

    definitions of existing MD arrays

    This file was auto-generated on Fri, 29 Oct 2010 18:16:55 +1000

    by mkconf 3.1.4-1+8efb9d1

    ARRAY /dev/md0 metadata=1.2 name=fog-2:0 UUID=7ca3e285:ced98dbe:5fb7d2f1:46804423
    [*]Array status

    [code]fog-2:~# cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : active raid0 sdb[0] sdc[1]
    3907026944 blocks super 1.2 256k chunks

    unused devices: <none>

    fog-2:~# mdadm --detail /dev/md0
    Version : 1.2
    Creation Time : Wed Jun 22 08:14:10 2011
    Raid Level : raid0
    Array Size : 3907026944 (3726.03 GiB 4000.80 GB)
    Raid Devices : 2
    Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Wed Jun 22 08:14:10 2011
          State : clean

    Active Devices : 2
    Working Devices : 2
    Failed Devices : 0
    Spare Devices : 0

    Chunk Size : 256K
          Name : fog-2:0  (local to host fog-2)
          UUID : 7ca3e285:ced98dbe:5fb7d2f1:46804423
        Events : 0
    Number  Major  Minor  RaidDevice State
      0      8      16        0      active sync  /dev/sdb
      1      8      32        1      active sync  /dev/sdc

    [*]Performance - 225.30 MB/sec - cached reads (from the drives cache) will be about 30-40x this speed.

    [code]fog-2:~# hdparm -t /dev/md0

    Timing buffered disk reads: 676 MB in 3.00 seconds = 225.30 MB/sec
    Please note that there is NO data redundancy in this setup. All images in this system are backup via DPM nightly.

    Watching each node with ‘vnstat -l -ru’ i can see that each node maxes out around [B]240-260 MB (megabytes)/sec[/B]. This setup easily uses the 2 hard drives to their full potential and would require faster disks and a hardware RAID controller to get more.

    This how-to will be left open for questions. Please let me know if you have any questions, or if i have missed something.

  • Developer


    After adding another storage node with the same configuration, but using newer 4k drives i managed to get a little more perforamance out of the setup.

    [code]fog-4:~# hdparm -t /dev/md0

    Timing buffered disk reads: 854 MB in 3.00 seconds = 284.22 MB/sec[/code]

  • Developer

    Actually, you are correct about full duplex. I have edited out the misinformation 🙂

    Maximum through put for 1x 1000BASE-X is listed at 125mb/sec.

    A VLAN isnt required, but will help separate traffic/broadcasts/etc

  • It is my understanding that Gigabit ports set to full duplex are 2Gbit total, 1Gb up and 1Gb down, which is why I was thinking I would only need 2 Gigabit Nics on the server and 2 ports on each switch for the trunk. I thought about VLans but in my environment that is adding one more thing that would be out of my control and there won’t be so much traffic going across all day that it would be an issue.

  • Developer

    I would be using 802.3ad, make sure both network cables are going into the same switch.
    Then i would create a VLAN between the switches to keep the traffic separate.

    It would be great if you could report any speeds back for comparison.

  • I have a question regarding bonding. My setup or soon to be setup is going to be the following, 2x Intel Gigabit NIC for the fog server with Raid 5 x6 10k hard drives, and 2 cisco gigabit switches (1 being a 2960G series) with 2 cables run between the switches for bonding/trunking/lacp. What would the best way be to set this upbe? The fog server will be in the server room connected to the first Cisco switch and then I will have 2 cables run to the 2960 switch to provide the room I’m in with 2Gb bandwidth. Where do I go from there, 802.3ad or 802.1q? Not much of a network guy so any help would be appreciated. My ultimate goal is to have 2Gb of bandwidth from the fog server to my room which is about 150ft away and where I do all my imaging.