IP over Infiniband network configuration best practices

Question

Hi EEC Team,
A question I've been asked a few times, do we have any best practices or ideas on how best to implement the IPoIB network?
Should it be Class B or C?
Also, what are your thoughts in regards to the netmask, if we use /24 it doesn't give us the ability to visually separate two different racks (ie Exalogic / Exadata), whereas netmask /23, we can do something like:
Exalogic : 192.168.*10*.0
Exadata : 192.168.*11*.0
While still being on the same subnet.
Your thoughts?
Gavin

Answer 1

I think it depends on a couple of factors, such as the following:
a) How many racks will be connected together on the same IPoIB fabric
b) What rack configuration do you have today, and do you foresee any expansion in the future - it is possible that you will move from a purely physical environment to a virtual environment, and you should consider the number of virtual hosts and their IP requirements when choosing a subnet mask.
Class C (/24) with 256 IP values is a good start. However, you may want to choose a mask of length 23 or even 22 to ensure that you have enough IPs for running the required number of WLS, OHS, Coherence Server instances on two or more compute nodes assigned to a department for running its application.
In general, when setting a net mask, it is always important that you consider such growth projections and possibilities.
By the way, in my view, Exalogic and Exadata need not be in the same IP subnet, especially if you want to separate application traffic from database traffic. Of course, they can be separated by VLANs too.
Hope this helps.
Thanks
Guru