About ASM and SAN...

Question

Hello Guys,
I have to implement 3 nodes RAC 10gR2 ob centOS4 operating system. I have study so many documents about rac instaltion and configurations. I have learn how to set the network requirements with private, public and virtual IPs and all other stuff. I have learn installtion of clusterware and database with cluster enable functionality.
BUT the storage options are still not clear to me. We have purchases SAN and we are planning to implement ASM for the storage. Now i want to know:
How many disk and disk partitions 3 node structure will require on SAN?
How ASM will access SAN, or you can say OS will access this shared storage?
Voting disk and OCR can not be store on sharted storage and need to be store on raw devices... what these raw device can be? How it can be access by all nodes?
Above three questions are disturbing me a lot. If they are clear to me the whole storage concept will be clear and i can implement RAC.
Please help me by answering the above 3 questions. I will be vert greatful to you.
Regards,
Imran

Answer 1

How many disk and disk partitions 3 node structure will require on SAN?
There's no real answer to that! With Oracle generally, RAC or no RAC, the answer to how many disks you should have is "as many as possible". Partitioning is really up to you, too, depending on what you find easiest to manage. If you have a single SAN array, for example, comprised of 15 disks that you choose to partition into three or four logical volumes so that you can call one 'data', one 'redo', one 'OS', and one 'other' -that's entirely up to you, since Oracle could care less how you partition, what you call them or how many of them there are. Moreover, everything on every partition is being striped across those 15 disks anyway, so who cares?
I think, however, you might be thinking of the RAC-specific issues of the voting disk and the Oracle Cluster Registry. If you were using a cluster file system, they could be just two files on the file system, about 120M in size between them. Since you are going to use ASM and these two elements can't be stored inside an ASM array, you'll have to create two raw partitions for this purpose. The rest you then chop up for ASM's use.
It is NOT true, incidentally, that "Voting disk and OCR can not be store on sharted storage". By definition, the voting disk and OCR must be on shared storage! Indeed, raw partitions, ASM arrays and cluster file systems are ALL shared storage technologies. It just so happens that those two files can't use ASM... but raw or cfs are fine.
A raw partition is not, of course, intrinsically 'shared storage'... but if it's a raw partition on your SAN, to which all three of your nodes are physically attached, then it is shareable. It's shareable simply because three nodes can see it. And because there's no file system there with exclusive and blocking file locks, what one node does to a raw partition doesn't stop another node accessing it simultaneously (which is the definition of shared storage, of course).
How will ASM access SAN? By you partitioning the SAN into a number of logical volumes, each one of which will be kept raw, and you then declaring each such volume as a candidate disk. You'll wrap all candidate disks up into an ASM disk group... and then Oracle will write to that disk group and hence through to the underlying logical volumes. Which comes back to the original question: how many logical volumes should you create out of, say, a 15 disk LUN on a SAN?
Depends, as I said, on a lot of things, but for example RAID5 runs best when there are either 5 or 9 disks in the array (or did when last I looked at an EMC Clariion SAN!). So if your underlying RAID technology was going to be RAID5, you might well create 3 5-disk logical volumes on the one LUN. To let ASM use all 15 disks, you'd then create a 3-disk diskgroup (where 1 ASM disk = 1 SAN logical volume). On the other hand, you might want to keep some disks back for future storage, in which case a 1-disk ASM diskgroup representing a single 9-disk logical volume might be the go, with the remaining 6 disks on the LUN available for future expansion.
It's a complicated topic, unfortunately. You're dealing with physical storage which is already abstracted into logical volumes and then abstracted even further by wrapping those logical volumes up into ASM disk groups. You balance performance, expandability, management convenience, your SAN vendor's optimisation tricks and so on... and hopefully come out with something that works for you!