With Linux, the open source OS [Operating System] becoming more popular every day with various flavors [distributions], Bright Enterprise Labs decided that it would be a good idea to point out one excellent feature that Linux offers for free – in all versions on the market. Software based RAID (Redundant Array of Independent Disks). No fancy add on with driver support needed, all you need is two or more storage connectors on your mainboard. Not only SATA is what you can use, even a mix of IDE and SATA drives is possible, although we don’t recommend it due to the very difference in bandwidth that both interfaces offer. Also, two IDE drives on the same port will suffer performance when they have to share the available bandwidth of that link.

Most of our readers will know the difference between RAID0, RAID1, RAID (1+0) aka. RAID10, RAID4, RAID5 and RAID6, but we’d like to briefly introduce them to you. The definition ‘n’ means the total capacity of a disk drive.

RAID0 a.k.a. Drive Span or Stripping (Total capacity = n * number of drives)
Requires two drives and will extend one drive to the next one. This will give you a maximum of capacity, where you reach 100% of your total capacity in one drive, but if one drive fails, all your data will be lost. Also, the reason why it is very popular is the fact that it speeds up your performance quite a bit because it can read / write to two disks at a time, thus doubling the performance of a single drive.
 
RAID1 a.k.a. Drive Mirror or Mirroring (Total capacity = total capacity / 2)
Requires two drives. All your data on drive #1 will be written instantly to drive #2. This will give you only 50% of your total capacity, but a 50:50 chance if a single drive fails; the remaining one will simply continue to handle your I/O requests.

RAID10 a.k.a. 0+1 (Spanned Mirror) (Total capacity = total capacity / 2)
Requires at least four drives [any number of drives that can be divided by two]. It adds the functionality of both RAID0 and RAID1. Two or more drives will be accessed as a span, but at the same time, requiring the same number of drives for a second span where all the data is also mirrored to. Also, you get the speedup that RAID0 offers. This will also give you 50% of your total capacity, but enables you to use more than 2 disks in one array.
 
RAID4 (Total capacity = n * number of drives – n)
Requires at least three disk drives. The major difference to other RAID levels is the fact that there is one dedicated drive containing parity data. Thus on every write among the data disks, there’s a write on the parity disk, causing a lot more write accesses to this disk and limiting the maximum RAID performance to the maximum performance of that parity disk. Also, due to the higher number of accesses, this drive is most likely to fail.

RAID5 (Total capacity = n * number of drives – n)
Requires at least three drives, but has no limits if you want to add more drives. The capacity of one drive is removed from the total capacity. The extra capacity is needed for the extra parity data that is being stored among all drives. If one drive fails, the RAID will become ‘degraded’ but will still let you read data off this drive. If another drive fails, all your data will be lost.

RAID6 (Total capacity = n * number of drives – 2 * n)
Requires four or more drives and is similar to RAID5, simply writing the parity data to two different drives and thus is able to handle a drive failure and still allows you to write – if a second drive fails, you’ll only be able to read from this drive. Think of it as RAID5+1.

There are many more RAID levels available. Sometimes vendor specific (in other words, Netapp has modified some RAID levels within its own product lineup and Intel uses RAID1 & RAID0 on its drive matrix option found in today’s chipsets) or simply not found frequently on the market, so we’ll stick to the most used RAID levels.

Linux RAID modules (md) also offers you to add one or more ‘spare’ drive to any RAID other than RAID0, where you do not have any safety and a spare drive doesn’t make sense at all. This spare drive will jump in and replace a failed disk immediately, so your system doesn’t run with a problem for too long.

A downside of Linux RAID is that while it is not impossible to make it work, it’s pretty tricky to boot from a RAID drive setup, so you should have a single drive to boot from, after that, the rest of the system can run off a RAID. But we’re sure that this will be solved in the future, where you’ll have the option of setting up a RAID at the time of Linux installation.

CONTINUED: Creating a RAID, Adding a drive, MDAM Growing RAID, Conclusion.


Creating a RAID array

Enough of the theory, here’s how you can create a drive RAID via the command line on pretty much any flavor of Linux. Now, just like every Linux, this is simple operation if you follow the instructions. Do not be afraid and just take things one step at the time. We are going to try to be as clear as possible, but this is not a walk in the park such as Windows shenanigans.

Our example is creation of a RAID array on a virtual machine with 6 virtual 100 MB disks. If you create a RAID5 among four or more SATA disks, the creation process can range from a few minutes up to an hour or more, depending on your system performance. We will take a look at what this procedure can do for your virtual machine in a follow-up story.

Initial window, preparing for RAID initialization
Initial window, preparing for RAID initialization

First thing you need to need are the mdadm tools that are used in this guide. Not all distributions supply this with a basic setup, so you‘d need either "apt-get install mdadm" if you’re on a Debian based Linux Distribution (like Ubuntu) or just check with your package manager to install the mdadm tools.

Second thing to do is to create a partition and mark it with a type that Linux will recognize as a special RAID partition.

admin@debian-vm:~$ sudo fdisk /dev/sdb                                                                                                                      

Command (m for help): m   (Type in ‘m’) (Let’s look at the options we have here)
Command action         
   a    toggle a bootable flag
   b    edit bsd disklabel
   c    toggle the dos compatibility flag
   d    delete a partition
   l     list known partition types
   m  print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t    change a partition’s system id
   u   change display/entry units
   v    verify the partition table
   w   write table to disk and exit
   x    extra functionality (experts only)    

Command action         
   e   extended        
   p   primary partition (1-4)
                         
Partition number (1-4): 1     (Type in ’1′) (We want the whole drive, so we need just one partition)
First cylinder (1-102, default 1): (simply press ENTER to accept the default values)
Using default value 1              
Last cylinder or +size or +sizeM or +sizeK (1-102, default 102):
Using default value 102                                          

Command (m for help): t    (Type in ‘t’) (Now we need to set the type of the partition)
Selected partition 1
   
Hex code (type L to list codes): L   (Type in ‘L’) (Just to check the options we have available)

 0  Empty           1e  Hidden W95 FAT1 80  Old Minix       be  Solaris boot   
 1  FAT12           24  NEC DOS         81  Minix / old Lin bf  Solaris        
 2  XENIX root      39  Plan 9          82  Linux swap / So c1  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  83  Linux           c4  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 5  Extended        41  PPC PReP Boot   85  Linux extended  c7  Syrinx         
 6  FAT16           42  SFS             86  NTFS volume set da  Non-FS data    
 7  HPFS/NTFS       4d  QNX4.x          87  NTFS volume set db  CP/M / CTOS / .
 8  AIX             4e  QNX4.x 2nd part 88  Linux plaintext de  Dell Utility   
 9  AIX bootable    4f  QNX4.x 3rd part 8e  Linux LVM       df  BootIt         
 a  OS/2 Boot Manag 50  OnTrack DM      93  Amoeba          e1  DOS access     
 b  W95 FAT32       51  OnTrack DM6 Aux 94  Amoeba BBT      e3  DOS R/O        
 c  W95 FAT32 (LBA) 52  CP/M            9f  BSD/OS          e4  SpeedStor      
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi eb  BeOS fs        
 f  W95 Ext’d (LBA) 54  OnTrackDM6      a5  FreeBSD         ee  EFI GPT        
10  OPUS            55  EZ-Drive        a6  OpenBSD         ef  EFI (FAT-12/16/
11  Hidden FAT12    56  Golden Bow      a7  NeXTSTEP        f0  Linux/PA-RISC b
12  Compaq diagnost 5c  Priam Edisk     a8  Darwin UFS      f1  SpeedStor      
14  Hidden FAT16 <3 61  SpeedStor       a9  NetBSD          f4  SpeedStor      
16  Hidden FAT16    63  GNU HURD or Sys ab  Darwin boot     f2  DOS secondary  
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fd  Linux raid auto
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fe  LANstep        
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid ff  BBT            
1c  Hidden W95 FAT3 75  PC/IX
Hex code (type L to list codes): fd    (Type in ‘fd’) (fd – ‘Linux raid auto’ is what we need)
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): p   (Type in ‘p’) (Have a look at the new partition table)

Disk /dev/sdb: 107 MB, 107413504 bytes
64 heads, 32 sectors/track, 102 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk identifier: 0x2c47d250

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         102      104432   fd  Linux raid autodetect

If you think something is wrong and decide to exit without making any changes, type in ‘q’

Command (m for help): w       (all fine, let’s write that partition table to the disk)
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
admin@debian-vm:~$

Now you’ll need to repeat that with all the drives you want to add to your RAID
example:
sudo fdisk /dev/sdc
sudo fdisk /dev/sdd

Taking it easy, just type in commands that suit your needs - all the commands are listed below.
Taking it easy, just type in commands that suit your needs – all the commands are listed below.

After this is done for all the disks in your system, you can easily add all your drives to the new RAID you want to have. All you need is this little line:

mdadm –create /dev/device_name (we use mdX) –level=Y (anything from 0, 1, 10, 4, 5 or 6) –raid-devices=Z (number of drives you want to add) /dev/hdX1 or /dev/sdX1 (you’ll need to know the names of the partitions you want to add and set that ’1′ because you want to access the first partition of that drive)

Creating a RAID0
admin@debian-vm:~$ sudo mdadm –create /dev/md0 –level=0 –raid-devices=2 /dev/sdb1 /dev/sdc1
mdadm: array /dev/md0 started.
admin@debian-vm:~$ cat /proc/mdstat     (this will give us a brief info on our RAIDs)
Personalities: [raid0]
md0 : active raid0 sdc1[1] sdb1[0]
      208640 blocks 64k chunks

unused devices: <none>

Creating a RAID1
admin@debian-vm:~$ sudo mdadm –create /dev/md0 –level=1 –raid-devices=2 /dev/sdb1 /dev/sdc1
mdadm: array /dev/md0 started.
admin@debian-vm:~$ cat /proc/mdstat     (this will give us a brief info on our RAIDs)
Personalities: [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
      104320 blocks 64k chunks

unused devices: <none>

Creating a RAID10
admin@debian-vm:~$ sudo mdadm –create /dev/md10 –level=10 –raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: array /dev/md10 started.
admin@debian-vm:~$ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10]
md10 : active raid10 sde1[3] sdd1[2] sdc1[1] sdb1[0]
      208640 blocks 64K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>
admin@debian-vm:~$

Creating a RAID4
admin@debian-vm:~$ sudo mdadm –create /dev/md4 –level=4 –raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: array /dev/md10 started.
admin@debian-vm:~$ cat /proc/mdstat    (so it’s running in the background, let’s check it)
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md4 : active raid4 sde1[4] sdd1[2] sdc1[1] sdb1[0]
      312960 blocks level 4, 64k chunk, algorithm 0 [4/3] [UUU_]
      [==============>......]  recovery = 72.5% (76544/104320) finish=0.0min speed=25514K/sec

unused devices: <none>
admin@debian-vm:~$ cat /proc/mdstat     (do it again to check if it’s done)
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md4 : active raid4 sde1[3] sdd1[2] sdc1[1] sdb1[0]
      312960 blocks level 4, 64k chunk, algorithm 0 [4/4] [UUUU]

unused devices: <none>
admin@debian-vm:~$

Creating a RAID5
admin@debian-vm:~$ sudo mdadm –create /dev/md5 –level=5 –raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: array /dev/md5 started.
admin@debian-vm:~$ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md5 : active raid5 sde1[4] sdd1[2] sdc1[1] sdb1[0]
      312960 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
      [================>....]  recovery = 83.3% (87552/104320) finish=0.0min speed=29184K/sec

unused devices: <none>
admin@debian-vm:~$ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md5 : active raid5 sde1[3] sdd1[2] sdc1[1] sdb1[0]
      312960 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

Creating a RAID6
admin@debian-vm:~$ sudo mdadm –create /dev/md6 –level=6 –raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: array /dev/md6 started.
admin@debian-vm:~$ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md6 : active raid6 sde1[3] sdd1[2] sdc1[1] sdb1[0]
      208640 blocks level 6, 64k chunk, algorithm 2 [4/4] [UUUU]
      [=======>.............]  resync = 39.2% (41600/104320) finish=0.0min speed=13866K/sec

unused devices: <none>
admin@debian-vm:~$ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md6 : active raid6 sde1[3] sdd1[2] sdc1[1] sdb1[0]
      208640 blocks level 6, 64k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

That’s fairly straight forward and easy. Now you can put on any kind of file system that you want to use. Ext2 or Ext3 are very common, but you can also use XFS or ReiserFS on this. It works just like a new hard drive. The name is the name you set in the /dev/mdX parameter used when the RAID was created.

Once you initiated the RAID creation process, it’ll automatically run in the background and you can view at the status by looking at "/proc/mdstat". Once the creation is done, you should consider checking this device very frequently or set up mdadm to send you emails once something happens. The man pages of mdadm are very well written and will tell you everything about the capabilities of the MD modules. We strongly recommend you take a look at them before you create a RAID.

 

CONTINUED: Adding a spare drive, Growing your RAID and Conclusion


Adding a spare drive to your RAID

admin@debian-vm:~$ sudo mdadm /dev/md6 -a /dev/sdf1
mdadm: added /dev/sdf1
admin@debian-vm:~$ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md6 : active raid6 sdf1[4](S) sde1[3] sdd1[2] sdc1[1] sdb1[0]
      208640 blocks level 6, 64k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

Here you can see that the drive sdf1 has now the status (S) next to it, marking it as a spare drive.

A very excellent feature that Linux RAID offers is the grow function. This will work best if you make use of the "Logical Volume Manager – short: LVM" within Linux so you can easily expand ext2 or ext3 or other types of file systems at ease. Beware, that growing a drive doesn’t equal growing its partition table, so you really need to keep that in mind.

Growing your RAID

admin@debian-vm:~$ sudo mdadm –grow /dev/md6 -n 6
mdadm: Need to backup 256K of critical section..
mdadm: … critical section passed.
admin@debian-vm:~$ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md6 : active raid6 sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
      417280 blocks level 6, 64k chunk, algorithm 2 [6/5] [UUUUU_]

unused devices: <none>
admin@debian-vm:~$

The MDADM tools provide you with a simple command line to check the status of your RAID:

either:

cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md6 : active raid6 sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
      312960 blocks level 6, 64k chunk, algorithm 2 [5/5] [UUUUU]

unused devices: <none>

or:

sudo mdadm –detail /dev/md6
/dev/md6:
        Version : 00.90
  Creation Time : Thu Apr  2 12:49:57 2009
     Raid Level : raid6
     Array Size : 312960 (305.68 MiB 320.47 MB)
  Used Dev Size : 104320 (101.89 MiB 106.82 MB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 6
    Persistence : Superblock is persistent

    Update Time : Thu Apr  2 12:50:04 2009
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

     Chunk Size : 64K

           UUID : 5cb84945:1c101341:47489f1c:03bfb314 (local to host debian-vm)
         Events : 0.4

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       3       8       65        3      active sync   /dev/sde1
       4       8       81        4      active sync   /dev/sdf1

Now, the reason for this tutorial is not to only show you how easy it is, but also to point out a few brilliant features that we really enjoy with software-based RAID configuration.

First: The ability to grow a RAID. This is rare to find on other RAID systems, where you usually would have to move all the data off your current drive, reconfigure it to the larger size and put the data back on. This will sadly, cause a long downtime. MDADM offers growth without you needing to move the data off and back on. Of course, we do recommend that you should make a backup anyways, but you’ll save the restore time and don’t have to reboot your whole system. All you need is to stop the services and applications that access your RAID’ed drive and unmount it, so you can grow it. To be honest, you can grow your RAID even without unmounting it, and people have documented that it works. But our official stand is that we do not recommend that – especially if you don’t have a backup of all your data. Save yourself the worries.

Second: Force a reassemble of a RAID. It might be unlikely, but it can happen quite easily. Two drives on your RAID5 fail because something stupid happens. Maybe you bump the case and the power-line split that you are using gets a short disconnect and your RAID is gone. Not all hope is lost. If you’re lucky, you might be able to force a reassemble and everything is working again. Sadly, when it comes to a hard-RAID configuration, unfortunately, most of RAID controllers don’t have this option – if your company has a product with the reassemble RAID option, let us know and we’ll be glad to test it.

This of course will only work if there was no data to be written to the disks at the time it happens. That’s why we recommend a Kernel setting called: deadline where you might lose some performance on the writing because write caching is disabled, but in case you don’t have a journaling file system (which would be stupid, too) it can save your data.

Third: If you run out of onboard ports for your RAID to be grown, you can plug in any SATA controller (starting with a two port up to any other kind of controller) and add those drives to your RAID as well.

Fourth: Once you understand the functionality of mdadm, you’ll find it astonishingly easy to use. It’s so simple.

Conclusion

If you are using Linux, we would reccomend this RAID method for all your data. It’s free, fast, simple and pretty well proven. In our follow-up article, coming in days following Easter Monday, we will take a close look at software RAID vs. hardware RAID pretty and show you some hard data on the performance and ease of setup.