How-to: Network bonding & RAID arrays
[I]Please note: This is an advanced topic and requires basic Linux knowledge. These examples only provide information on how my system was configured.[/I]
Using the following setup i can unicast 15-20 machines at full speed (per node, 40 in total with 2 nodes). Unicast speeds range from 5.5gb/minute (~90mb/sec) to 3gb/minute (~50mb/sec). Normally it is much faster to unicast over multicast due to multicast’s fiddly deployments.
[SIZE=4][B]FOG VM Controller[/B] - FOG-1[/SIZE]
Stores no images, only used for a controller
[SIZE=4][B]FOG Storage Nodes[/B] - FOG-2 & FOG-3[/SIZE]
These machines are custom white boxes, they were left over workstations which were turned into FOG storage nodes. Nothing fancy here.
CPU: E8400 Core2Duo
Motherboard: Intel DG35EC
RAM: DDR2 4gb - overkill really
OS Drive: 80gb Western Digital Green - Image storage is on the RAID array below
2x Intel 82574L PCIe 1gb network cards - bonded
Uses package ‘ifenslave’ (apt-get install ifenslave)
Bond Mode 0 - mode 4 (802.3ad) works [I]slightly[/I] better, but requires switch support
eth1 & 2
eth0 still has a cable in case something happens to the bond, the server can still be accessed remotely
[code]fog-2:~# cat /etc/network/interfaces
This file describes the network interfaces available on your system
and how to activate them. For more information, see interfaces(5).
The loopback network interface
iface lo inet loopback
The primary network interface
iface eth0 inet dhcp
bonded network connections
iface bond0 inet static
slaves eth1 eth2
pre-up ifdown eth0
post-down ifup eth0
[*]module setup - not sure if this is still required with the new ifenslave method
[code]fog-2:~# cat /etc/modprobe.d/arch-aliases
alias bond0 bonding
options bonding mode=0 miimon=100
[*]ifconfig (lo removed)
bond0 Link encap:Ethernet HWaddr 00:1b:21:53:c0:e5
inet addr:10.10.8.55 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::21b:21ff:fe53:c0e5/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:11906 errors:0 dropped:0 overruns:0 frame:0
TX packets:6105 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:2167156 (2.0 MiB) TX bytes:1736174 (1.6 MiB)
eth1 Link encap:Ethernet HWaddr 00:1b:21:53:c0:e5
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:10145 errors:0 dropped:0 overruns:0 frame:0
TX packets:3053 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:1912960 (1.8 MiB) TX bytes:869040 (848.6 KiB)
eth2 Link encap:Ethernet HWaddr 00:1b:21:53:c0:e5
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1761 errors:0 dropped:0 overruns:0 frame:0
TX packets:3052 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:254196 (248.2 KiB) TX bytes:867134 (846.8 KiB)
2x 2TB Western Digital Black
Non raid edition drives, just the standard blacks
Software RAID using MDADM
Create file systems
[code]mke2fs -t ext4 -b 4096 -m 0 -i 65536 -E stride=64,stripe-width=128 /dev/md0[/code]
Create a partition on md0 as EXT4, 4k block sizes, no root reserved space, 65k inode block sizes
The images will be 1 big massive block of data. Maximizing block and inode sizes reduces overhead at the expense of a little wasted space (if the file partly uses a block).
stride = This is the number of blocks read or written to disk before moving to next disk
stripe_width = This is typically be stride-size * N, where N is the number of data disks in the RAID
If you are unsure of how to calculate stride and stripe_width, there is a handy calculator here: [url]http://busybox.net/~aldot/mkfs_stride.html[/url]
[*]Create MDADM Array
chunk size = 256kb # this could be 512kb, but i have seen no performance increase on this hardware set
A larger chunk size results in less overhead (faster performance) at the expense of wasted disk. The amount of waste will be minimal was we are using EXT4 with preallocate enabled. EXT preallocate has no overhead, where EXT3 must manually zero the files. Get EXT4!!
[code]mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=2 --chunk=256 /dev/sdb1 /dev/sdc1[/code]
fog-2:~# mdadm --detail --scan >> /etc/mdadm/mdadm.conf
fog-2:~# cat /etc/mdadm/mdadm.conf
Please refer to mdadm.conf(5) for information about this file.
by default, scan all partitions (/proc/partitions) for MD superblocks.
alternatively, specify devices to scan, using wildcards if desired.
auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
automatically tag new arrays as belonging to the local system
instruct the monitoring daemon where to send mail alerts
definitions of existing MD arrays
This file was auto-generated on Fri, 29 Oct 2010 18:16:55 +1000
by mkconf 3.1.4-1+8efb9d1
ARRAY /dev/md0 metadata=1.2 name=fog-2:0 UUID=7ca3e285:ced98dbe:5fb7d2f1:46804423
[code]fog-2:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid0 sdb sdc
3907026944 blocks super 1.2 256k chunks
unused devices: <none>
fog-2:~# mdadm --detail /dev/md0
Version : 1.2
Creation Time : Wed Jun 22 08:14:10 2011
Raid Level : raid0
Array Size : 3907026944 (3726.03 GiB 4000.80 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Wed Jun 22 08:14:10 2011 State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Chunk Size : 256K Name : fog-2:0 (local to host fog-2) UUID : 7ca3e285:ced98dbe:5fb7d2f1:46804423 Events : 0 Number Major Minor RaidDevice State 0 8 16 0 active sync /dev/sdb 1 8 32 1 active sync /dev/sdc
[*]Performance - 225.30 MB/sec - cached reads (from the drives cache) will be about 30-40x this speed.
[code]fog-2:~# hdparm -t /dev/md0
Timing buffered disk reads: 676 MB in 3.00 seconds = 225.30 MB/sec
Please note that there is NO data redundancy in this setup. All images in this system are backup via DPM nightly.
Watching each node with ‘vnstat -l -ru’ i can see that each node maxes out around [B]240-260 MB (megabytes)/sec[/B]. This setup easily uses the 2 hard drives to their full potential and would require faster disks and a hardware RAID controller to get more.
This how-to will be left open for questions. Please let me know if you have any questions, or if i have missed something.
After adding another storage node with the same configuration, but using newer 4k drives i managed to get a little more perforamance out of the setup.
[code]fog-4:~# hdparm -t /dev/md0
Timing buffered disk reads: 854 MB in 3.00 seconds = 284.22 MB/sec[/code]
Actually, you are correct about full duplex. I have edited out the misinformation :)
Maximum through put for 1x 1000BASE-X is listed at 125mb/sec.
A VLAN isnt required, but will help separate traffic/broadcasts/etc
Charles last edited by
It is my understanding that Gigabit ports set to full duplex are 2Gbit total, 1Gb up and 1Gb down, which is why I was thinking I would only need 2 Gigabit Nics on the server and 2 ports on each switch for the trunk. I thought about VLans but in my environment that is adding one more thing that would be out of my control and there won’t be so much traffic going across all day that it would be an issue.
I would be using 802.3ad, make sure both network cables are going into the same switch.
Then i would create a VLAN between the switches to keep the traffic separate.
It would be great if you could report any speeds back for comparison.
Charles last edited by
I have a question regarding bonding. My setup or soon to be setup is going to be the following, 2x Intel Gigabit NIC for the fog server with Raid 5 x6 10k hard drives, and 2 cisco gigabit switches (1 being a 2960G series) with 2 cables run between the switches for bonding/trunking/lacp. What would the best way be to set this upbe? The fog server will be in the server room connected to the first Cisco switch and then I will have 2 cables run to the 2960 switch to provide the room I’m in with 2Gb bandwidth. Where do I go from there, 802.3ad or 802.1q? Not much of a network guy so any help would be appreciated. My ultimate goal is to have 2Gb of bandwidth from the fog server to my room which is about 150ft away and where I do all my imaging.