Storage Array 2530 - 36 @ 2TB HDD - RAID6

Question

Hello!
I have my Array up and running using the CAM software but I have a concern and I have little experience and trouble finding documentation on the issues..
I have a 36 SATA disk 2TB setup, with 3 hot spares. 1x3
My first issue is when creating the Volume/Virtual Disk Etc. It only seems to allow me to use 30 disk drives even though the other 3 are listed as Unassigned.. Is there a limitation to Raid 6 I am unaware of?
My second concern is the initialization time of about 510 Hours.. the job is increasing about 3% per day, does this seem right? The array is not reporting any issues and the firmware was updated.
Again, your support and help is always appreciated.
Regards and Thanks.
Ash
REHL 6.2 RD240 64 bit

Answer 1

On a file system that big?
Use a volume manager (LVM, ZFS, etc) to spread the file system across multiple RAID arrays. 10-drive RAID-6 arrays are good - 8 data, 2 parity. Pick a segment size for each array such that the RAID stripe size (segment size times number of data drives in array) is no larger than the underlying block size of the file or volume management system you plan to put on it. Multiple underlying RAID arrays will also allow both controllers of the 2530 (assuming it has two) to be used for processing IO, doubling both the effective processing power of the 2530 and the bandwidth to the device since your server will be able to actively use SAN paths to both controllers.
For example, if you use ZFS with its 128k block size, on 8+2 RAID-6 arrays, pick a 16k segment size so you get a 128k RAID stripe width. If you're using ext3 on top of LVM, select a segment size, calculate your stripe width, then use the -datalignment option of "pvcreate" to match your stripe width. Build an even number of RAID arrays, with an equal number on each controller.
If you manually have to partition/label each LUN, make sure you start the disk partition you're going to be writing data to on a RAID stripe boundary. For example, note that ZFS, when given a whole disk/LUN to use, puts an EFI label on the disk and starts its data partition at segment 256, which is at the first 128k boundary, assuming a 512-byte segment size.
On a file system this big, if data availability is critical, I'd also be tempted to use software RAID on top of the multiple RAID arrays. You'll lose a small bit of performance, but it might be worth it. In my experience, block size, segment size, and RAID stripe calculations are nowhere near as important for software RAID as they are for hardware RAID.
It's all trade-offs. The more drives you wind up using for parity, the more available your data will be.
Don't mess around with multiple LUNs off any physical RAID array in your 2530. Just fill the entire RAID array with a single LUN.
To get the best possible availability, if you're using Oracle's CAM to set up the RAID array, don't let it magically select which physical drives to put in each RAID array you build. If you have enough expansion trays, you want to minimize the number of drives you select from each tray. If you have a total of 5 trays and are building 10-drive RAID-6 arrays, for example, only pick two drives from each tray for each RAID array you build. That way, if an entire tray fails (someone goes into the server room and unplugs the wrong device...), you'll minimize the impact and won't lose access to your file system. If you don't have enough expansion trays to build your RAID arrays that way, try to build each RAID array in as few trays as possible. That way a tray failure will have a minimum impact on your file system (hopefully).
And don't forget - RAID is for availability. Backups are for integrity. RAID can't protect you from someone making a mistake and running "rm -f -r ." in the wrong place.