How-to: Network bonding & RAID arrays

Blackout

[I]Please note: This is an advanced topic and requires basic Linux knowledge. These examples only provide information on how my system was configured.[/I]

Using the following setup i can unicast 15-20 machines at full speed (per node, 40 in total with 2 nodes). Unicast speeds range from 5.5gb/minute (~90mb/sec) to 3gb/minute (~50mb/sec). Normally it is much faster to unicast over multicast due to multicast’s fiddly deployments.

[SIZE=4][B]FOG VM Controller[/B] - FOG-1[/SIZE]
[LIST]
[]Stores no images, only used for a controller
[]Debian
[]5gb HDD
[]1gb RAM
[/LIST]
[SIZE=4][B]FOG Storage Nodes[/B] - FOG-2 & FOG-3[/SIZE]
[LIST]
[][B]General[/B]
[LIST]
[]These machines are custom white boxes, they were left over workstations which were turned into FOG storage nodes. Nothing fancy here.
[]CPU: E8400 Core2Duo
[]Motherboard: Intel DG35EC
[]RAM: DDR2 4gb - overkill really
[]OS Drive: 80gb Western Digital Green - Image storage is on the RAID array below
[/LIST]
[][B]Network[/B]
[LIST]
[]2x Intel 82574L PCIe 1gb network cards - bonded
[]Uses package ‘ifenslave’ (apt-get install ifenslave)
[]Bond Mode 0 - mode 4 (802.3ad) works [I]slightly[/I] better, but requires switch support
[]eth1 & 2
[]eth0 still has a cable in case something happens to the bond, the server can still be accessed remotely
[*]interfaces

[code]fog-2:~# cat /etc/network/interfaces

This file describes the network interfaces available on your system

and how to activate them. For more information, see interfaces(5).

The loopback network interface

auto lo
iface lo inet loopback

The primary network interface

iface eth0 inet dhcp

[code]fog-2:~# cat /etc/modprobe.d/arch-aliases

network bonding

alias bond0 bonding
options bonding mode=0 miimon=100
[/code]
[*]ifconfig (lo removed)

[code]fog-2:~# ifconfig
bond0 Link encap:Ethernet HWaddr 00:1b:21:53:c0:e5
inet addr:10.10.8.55 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::21b:21ff:fe53:c0e5/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:11906 errors:0 dropped:0 overruns:0 frame:0
TX packets:6105 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2167156 (2.0 MiB) TX bytes:1736174 (1.6 MiB)

eth1 Link encap:Ethernet HWaddr 00:1b:21:53:c0:e5
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:10145 errors:0 dropped:0 overruns:0 frame:0
TX packets:3053 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1912960 (1.8 MiB) TX bytes:869040 (848.6 KiB)
Memory:e3280000-e32a0000

eth2 Link encap:Ethernet HWaddr 00:1b:21:53:c0:e5
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1761 errors:0 dropped:0 overruns:0 frame:0
TX packets:3052 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:254196 (248.2 KiB) TX bytes:867134 (846.8 KiB)
Memory:e3180000-e31a0000[/code]
[/LIST]
[][B]Storage[/B]
[LIST]
[]2x 2TB Western Digital Black
[]Non raid edition drives, just the standard blacks
[]RAID 0
[]Software RAID using MDADM
[]Create file systems

[code]mke2fs -t ext4 -b 4096 -m 0 -i 65536 -E stride=64,stripe-width=128 /dev/md0[/code]

Create a partition on md0 as EXT4, 4k block sizes, no root reserved space, 65k inode block sizes

The images will be 1 big massive block of data. Maximizing block and inode sizes reduces overhead at the expense of a little wasted space (if the file partly uses a block).

stride = This is the number of blocks read or written to disk before moving to next disk
stripe_width = This is typically be stride-size * N, where N is the number of data disks in the RAID

If you are unsure of how to calculate stride and stripe_width, there is a handy calculator here: [url]http://busybox.net/~aldot/mkfs_stride.html[/url]
[*]Create MDADM Array

chunk size = 256kb # this could be 512kb, but i have seen no performance increase on this hardware set

A larger chunk size results in less overhead (faster performance) at the expense of wasted disk. The amount of waste will be minimal was we are using EXT4 with preallocate enabled. EXT preallocate has no overhead, where EXT3 must manually zero the files. Get EXT4!!

[code]mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=2 --chunk=256 /dev/sdb1 /dev/sdc1[/code]

[*]MDADM config

[code]

Generated with:

fog-2:~# mdadm --detail --scan >> /etc/mdadm/mdadm.conf

fog-2:~# cat /etc/mdadm/mdadm.conf

mdadm.conf

Please refer to mdadm.conf(5) for information about this file.

by default, scan all partitions (/proc/partitions) for MD superblocks.

alternatively, specify devices to scan, using wildcards if desired.

DEVICE partitions

auto-create devices with Debian standard permissions

CREATE owner=root group=disk mode=0660 auto=yes

automatically tag new arrays as belonging to the local system

HOMEHOST <system>

instruct the monitoring daemon where to send mail alerts

MAILADDR root

definitions of existing MD arrays

This file was auto-generated on Fri, 29 Oct 2010 18:16:55 +1000

by mkconf 3.1.4-1+8efb9d1

ARRAY /dev/md0 metadata=1.2 name=fog-2:0 UUID=7ca3e285:ced98dbe:5fb7d2f1:46804423
[/code]
[*]Array status

[code]fog-2:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid0 sdb[0] sdc[1]
3907026944 blocks super 1.2 256k chunks

unused devices: <none>

fog-2:~# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Wed Jun 22 08:14:10 2011
Raid Level : raid0
Array Size : 3907026944 (3726.03 GiB 4000.80 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Wed Jun 22 08:14:10 2011
      State : clean

Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Chunk Size : 256K

      Name : fog-2:0  (local to host fog-2)
      UUID : 7ca3e285:ced98dbe:5fb7d2f1:46804423
    Events : 0

Number  Major  Minor  RaidDevice State
  0      8      16        0      active sync  /dev/sdb
  1      8      32        1      active sync  /dev/sdc

[/code]
[*]Performance - 225.30 MB/sec - cached reads (from the drives cache) will be about 30-40x this speed.

[code]fog-2:~# hdparm -t /dev/md0

/dev/md0:
Timing buffered disk reads: 676 MB in 3.00 seconds = 225.30 MB/sec
[/code]
[/LIST]
[/LIST]
Please note that there is NO data redundancy in this setup. All images in this system are backup via DPM nightly.

Watching each node with ‘vnstat -l -ru’ i can see that each node maxes out around [B]240-260 MB (megabytes)/sec[/B]. This setup easily uses the 2 hard drives to their full potential and would require faster disks and a hardware RAID controller to get more.

This how-to will be left open for questions. Please let me know if you have any questions, or if i have missed something.

Charles

I have a question regarding bonding. My setup or soon to be setup is going to be the following, 2x Intel Gigabit NIC for the fog server with Raid 5 x6 10k hard drives, and 2 cisco gigabit switches (1 being a 2960G series) with 2 cables run between the switches for bonding/trunking/lacp. What would the best way be to set this upbe? The fog server will be in the server room connected to the first Cisco switch and then I will have 2 cables run to the 2960 switch to provide the room I’m in with 2Gb bandwidth. Where do I go from there, 802.3ad or 802.1q? Not much of a network guy so any help would be appreciated. My ultimate goal is to have 2Gb of bandwidth from the fog server to my room which is about 150ft away and where I do all my imaging.

Blackout

I would be using 802.3ad, make sure both network cables are going into the same switch.
Then i would create a VLAN between the switches to keep the traffic separate.

It would be great if you could report any speeds back for comparison.

Charles

It is my understanding that Gigabit ports set to full duplex are 2Gbit total, 1Gb up and 1Gb down, which is why I was thinking I would only need 2 Gigabit Nics on the server and 2 ports on each switch for the trunk. I thought about VLans but in my environment that is adding one more thing that would be out of my control and there won’t be so much traffic going across all day that it would be an issue.

Blackout

Actually, you are correct about full duplex. I have edited out the misinformation

Maximum through put for 1x 1000BASE-X is listed at 125mb/sec.

A VLAN isnt required, but will help separate traffic/broadcasts/etc

Blackout

[B][SIZE=4]UPDATE[/SIZE][/B]

After adding another storage node with the same configuration, but using newer 4k drives i managed to get a little more perforamance out of the setup.

[code]fog-4:~# hdparm -t /dev/md0

/dev/md0:
Timing buffered disk reads: 854 MB in 3.00 seconds = 284.22 MB/sec[/code]