Teamed NICs for RAC interconnect

Hi there,
We have an Oralce 10g RAC with 2 nodes. there are only one NIC for RAC interconnect in both servers.
now we want to add one redundant NIC into each server for RAC interconnect as well.
Could you please guide me some documents about this "teamed NICs for RAC interconnect "?
Your help is greatly appreciated!
Thanks,
Scott

Search around for NIC bonding. The exact process will depend on your OS.
Linux, see Metalink note 298891.1 - Configuring Linux for the Oracle 10g VIP or private interconnect using bonding driver
Regards,
Greg Rahn
http://structureddata.org

Similar Messages

Dedicated switches needed for RAC interconnect or not?

Currently working on an Extended RAC cluster design implementation, I asked the network engineer for dedicated switches for the RAC interconnects.
Here is a little background:
There are 28 RAC clusters over 2X13 physical RAC nodes with separate Oracle_Home for each instance with atleast 2+ instances on each RAC node. So 13 RAC nodes will be in each site(Data-Center). This is basically an Extended RAC solution for SAP databases on RHEL 6 using ASM and Clusterware for Oracle 11gR2. The RAC nodes are Blades in a c7000 enclosure (in each site). The distance between the sites is 55+ kms.
Oracle recommends to have Infiniband(20GBps) as the network backbone, but here DWDM will be used with 2X10 Gbps (each at 10 GBps) links for the RAC interconnect between the sites. There will be separate 2x1GBps redundant link for the Production network and 2x2 GBps FC(Fiber-Channel) redundant links for the SAN/Storage(ASM traffic will go here) network. There will be switches for the Public-production network and the SAN network each.
Oracle recommends dedicated switches(which will give acceptable latency/bandwith) with switch redundancy to route the dedicated/non-routable VLANs for the RAC interconnect (private/heartbeat/global cache transfer) network. Since the DWDM interlinks is 2x10Gbps - do I still need the dedicated switches?
If yes, then how many?
Your inputs will be greatly appreciated.. and help me take a decision.
Many Thanks in advance..
Abhijit

Absolutely agree.. the chances of overload in a HA(RAC) solution and ultmate RAC node eviction are very high(with very high latency) and for exactly this reason I even suggested inexpensive switches to route the VLANs for the RAC interconnect through these switches. The ASM traffic will get routed through the 2x2GB FC links through SAN-Directors (1 in each site).
Suggested the network folks to use Up-links from the c7000 enclosure and route the RAC VLAN through these inexpensive switches for the interconnect traffic. We have another challenge here: HP has certified using VirtualConnect/Flex-Fabric architecture for Blades in c7000 to allocate VLANs for RAC interconnect. But this is only for one site, and does not span Production/DR sites separated over a distance.
Btw, do you have any standard switch model to select from.. and how many to go for a RAC configuration of 13 Extended RAC clusters with each cluster hosting 2+ RAC instances to host total of 28 SAP instances.
Many Thanks again!
Abhijit

NICs for Private Interconnect redundancy

DB/Grid version : 11.2.0.2
Platform : AIX 6.1
We are going to install a 2-node RAC on AIX (that thing which is almost good as Solaris )
Our primary private interconnect is
### Primary Private Interconnect
169.21.204.1      scnuprd186-privt1.mvtrs.net scnuprd186-privt1
169.21.204.4      scnuprd187-privt1.mvtrs.net scnuprd187-privt1For Cluster inteconnect's redundancy , Unix team has attached an extra NIC for each node with an extra Gigabit-ethernet switch for these NICs.
###Redundant Private Interconnect attached to the server
169.21.204.2      scnuprd186-privt2.mvtrs.net scnuprd186-privt2 # Node1's newly attached redundant NIC
169.21.204.5      scnuprd187-privt2.mvtrs.net scnuprd187-privt2 # Node2's newly attached redundant NICExample borrowed from citizen2's post
Apparently I have 2 ways to implement cluster inteconnect's redundancy
Option1. NIC bonding at OS level
Option2. Let grid software do it
Question1. Which is better : Option 1 or 2 ?
Question2.
Regarding Option2.
From googling and OTN , i gather that , during grid installation you just provide 169.21.204.0 for cluster inteconnect and grid will identify the redundant NIC and switch. And if something goes wrong with the Primary Interconnect setup (shown above) , grid will automatically re-route interconnect traffic using the redundant NIC setup. Is this correct ?
Question 3.
My colleague tells me , for the redundant Switch (Gigabit) Unless I configure some Multicasting (AIX specific), I could get errors during installation. He doesn't clearly what it was ? Anyone faced Multicasting related issue on this ?

Hi,
My recommendation is to you use the AIX EtherChannel.
The EtherCannel of AIX is much more powerfull and stable compared with HAIP.
See how setup AIX EtherChannel on 10 Gigabit Ethernet interfaces
http://levipereira.wordpress.com/2011/01/26/setting-up-ibm-power-systems-10-gigabit-ethernet-ports-and-aix-6-1-etherchannel-for-oracle-rac-private-interconnectivity/
If you choose use HAIP I recommend you read this note, and find all notes about bugs of HAIP on AIX.
11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip [ID 1210883.1]
ASM Crashes as HAIP Does not Failover When Two or More Private Network Fails [ID 1323995.1]
About Multicasting read it:
Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1]
Regards,
Levi Pereira

802.3ad (mode=4) bonding for RAC interconnects

Is anyone using 802.3ad (mode=4) bonding for their RAC interconnects? We have five Dell R710 RAC nodes and we're trying to use the four onboard Broadcom NetXtreme II NICs in a 802.3ad bond with src-dst-mac load balancing. Since we have the hardware to pull this off we thought we'd give it a try and achieve some extra bandwith for the interconnect rather than deploying the traditional acitve/standby interconnect using just two of the NICs. Has anyone tried this config and what was the outcome? Thanks.

I don't but may be the documents might help ?
http://www.iop.org/EJ/article/1742-6596/119/4/042015/jpconf8_119_042015.pdf?request-id=bcddc94d-7727-4a8a-8201-4d1b837a1eac
http://www.oracleracsig.org/pls/apex/Z?p_url=RAC_SIG.download_my_file?p_file=1002938&p_id=1002938&p_cat=documents&p_user=nobody&p_company=994323795175833
http://www.oracle.com/technology/global/cn/events/download/ccb/10g_rac_bp_en.pdf
Edited by: Hub on Nov 18, 2009 10:10 AM

RAC Interconnect Transfer rate vs NIC's Bandwidth

Hi Guru,
I need some clarification for RAC interconnect terminology between "private interconnect transfer rate" and "NIC bandwidth".
We have 11gR2 RAC with multiple databases.
So we need to find out what the current resource status is.
We have two physical NICs each node. And 8G is for public and 2G is for private (interconnect).
Technically, we have 4G for Private network bandwidth.
If I look at the "Private Interconnect Transfer rate" though OEM or IPTraf (linux tool), it is showing 20 ~30 MB/Sec.
There is no any issue at all at this moment.
Please correct me if I am wrong.
The transfer rate will be fine till 500M or 1G/Sec. Because the current NIC's capacity is 4G. Does it make sense ?
I'm sure there are multiple things to consider,but I'm kind of stumped on the whole transfer rate vs bandwidth. Is there any way to calculate what a typical transfer would be....
OR How do I say our interconnect are good enough ....based on the transfer rate ?
Another question is ....
In our case, how do I set up the warning threshold and Critical threshold for "Private Interconnect Transer rate" in OEM ?
Any comments will be appreciated.
Please advise.

Interconnect performance sways more to latency than bandwidth IMO. In simplistic terms, memory is shared across the Interconnect. What is important for accessing memory? The size of the pipe? Or the speed of the pipe?
A very fast small pipe will typically perform significantly better than a large and slower pipe.
Even the size of the pipe is not that straight forward. Standard IP MTU size is 1500. You can run jumbo and super-jumbo frame MTU sizes on the Interconnect - where for example a MTU size of 65K is significantly larger than a 1500 byte MTU. Which means significantly more data can be transferred over the Interconnect at a much reduced overhead.
Personally, I would not consider Ethernet (GigE included) for the Interconnect. Infiniband is faster, more scalable, and offers an actual growth path to 128Gb/s and higher.
Oracle also uses Infiniband (QDR/40Gb) for their Exadata Database Machine product's Interconnect. Infiniband also enables one to run Oracle Interconnect over RDS instead of UDP. I've seen Oracle reports to the OFED committee saying that using RDS in comparison with UDP, reduced CPU utilisation by 50% and decreased latency by 50%.
I also do not see the logic of having a faster public network and a slower Interconnect.
IMO there are 2 very fundamental components in RAC that determines what is the speed and performance achievable with that RAC - the speed, performance and scalability of the I/O fabric layer and for the Interconnect layer.
And Exadata btw uses Infiniband for both these critical layers. Not fibre. Not GigE.

NIC teaming/bonding for interconnection

Hi to all,
We are planning for a two node Oracle RAC implementation on Windows Server and we have some doubts about interconnect. Because redundancy is not as important, and because we are limited by money Gigabit switch will be used for interconnection and for public network.
Can we get in bandwidth (load balancing) if we use NIC Bonding on one switch. Which is best?
- To buy servers with two single 1Gb port NIC’s and bond this NIC’s for interconnection and one single port NIC for public network or
- One dual port NIC (bond this NIC’s) for interconnects and one single NIC for public network?

Nikoline wrote:
We are planning for a two node Oracle RAC implementation on Windows Server and we have some doubts about interconnect. Because redundancy is not as important, and because we are limited by money Gigabit switch will be used for interconnection and for public network.Keep in mind that the h/w architecture used, directly determines how robust RAC is, how well it performs, and how it meets redundancy, high availability and scalability requirements.
The wrong h/w choices can and will hurt all of these factors.
Can we get in bandwidth (load balancing) if we use NIC Bonding on one switch. No. Bonding usually does not address load balancing. You need to check exactly what the driver stack used for bonding supports.
The primary reason for bonding is redundancy. Not load balancing/load sharing.
Which is best?
- To buy servers with two single 1Gb port NIC’s and bond this NIC’s for interconnection and one single port NIC for public network or
- One dual port NIC (bond this NIC’s) for interconnects and one single NIC for public network?A server with dual 1Gb NICs will be bonded into a single bonded NIC. And that is what then must be used - the logical bonded NIC and not the individual NICs.
This means that 2 1Gb ports provide you with a single bonded port. And you need 2 ports for RAC - public and interconnect.
So you either need 4 physical ports in total to create 2 bonded ports, or 3 ports in total for 1 bonded port and 1 unbonded port.
And what about your cluster's shared storage?
This, together with the Interconnect, determines the robustness and performance of RAC. 1Gb Interconnect is already pretty much a failure in providing a proper Interconnect infrastructure for RAC.
Keep in mind that the Interconnect is used to share memory across cluster nodes (cache fusion). The purpose is to speed up I/O - making it faster for a cluster node to get data blocks from another node's cache instead of having to hit spinning rust to read that data.
Old style fibre channel technology for shared storage is dual 2Gb fibre channels. Which will be faster than your 1Gb Interconnect. How does it make sense to use an Interconnect that is slower (less bandwidth and more latency) than the actual I/O fabric layer?
Would you configure h/w for a Windows server (be that a database, web or mail server) where the logical I/O from the server's memory buffer cache for local disks is slower than actually reading that data off local disk?
Then why do this for RAC?
Last comment - over 90% of the 500 fastest and biggest clusters on this planet run Linux. 1% runs Windows. Windows is usually a poor choice when it comes to clustering. Even Oracle does not provide their Exadata Database Machine product (fastest RAC cluster in the world) on Windows - only on Linux and Solaris (and the latter only because Oracle now owns Solaris too).

What is acceptable level of Private Interconnect Latency for RAC

We have build 3 Node RAC on RHEL5.4 on VMware.
There is node eviction problem due to loss of Network Heartbeat.
ocssd.log:[ CSSD]2010-03-05 17:48:21.908 [84704144] >TRACE: clssnmReadDskHeartbeat: node 3, vm-lnx-rds1173, has a disk HB, but no network HB, DHB has rcfg 0, wrtcnt, 2, LATS 1185024, lastSeqNo 2, timestamp 1267791501/1961474
Ping statistics from Node2 to Node1 are as below
--- rds1171-priv ping statistics ---
443 packets transmitted, 443 received, 0% packet loss, time 538119ms
rtt min/avg/max/mdev = 0.150/2.030/630.212/29.929 ms
[root@vm-lnx-rds1172 oracle]#
Can this be reason for Node eviction? What is acceptable level of of private interconnect latency for RAC ?

What is acceptable level of of private interconnect latency for RAC ?Normal local network latency should be enough. By the way latency settings are very generous.
Can you check if your to-be-evicted node runs and is reachable when seeing the node eviction messages?
In addition to that: Can you check the log files of the eviced node. Check for time stamps around "2010-03-05 17:48:21.908". Make sure all systems are NTP synchronized.
Ronny Egner
My Blog: http://blog.ronnyegner-consulting.de

Slow migration rates for shared-nothing live migration over teaming NICs

I'm trying to increase the migration/data transfer rates for shared-nothing live migrations (i.e., especially the storage migration part of the live migration) between two Hyper-V hosts. Both of these hosts have a dedicated teaming interface (switch-independent,
dynamic) with two 1GBit/s NICs which is used for only for management and transfers. Both of the NICs for both hosts have RSS enabled (and configured), and the teaming interface also shows RSS enabled, as does the corresponding output from Get-SmbMultichannelConnection).
I'm currently unable to see data transfers of the physical volume of more than around 600-700 MBit/s, even though the team is able to saturate both interfaces with data rates going close to the 2GBit/s boundary when transferring simple files over SMB. The
storage migration seems to use multichannel SMB, as I am able to see several connections all transferring data on the remote end.
As I'm not seeing any form of resource saturation (neither the NIC/team is full, nor is a CPU, nor is the storage adapter on either end), I'm slightly stumped that live migration seems to have a built-in limit to 700 MBit/s, even over a (pretty much) dedicated
interface which can handle more traffic when transferring simple files. Is this a known limitation wrt. teaming and shared-nothing live migrations?
Thanks for any insights and for any hints where to look further!

Compression is not configured on the live migrations (but rather it's set to SMB), but as far as I understand, for the storage migration part of the shared-nothing live migration this is not relevant anyway.
Yes, all NICs and drivers are at their latest version, and RSS is configured (as also stated by the corresponding output from Get-SmbMultichannelConnection, which recognizes RSS on both ends of the connection), and for all NICs bound to the team, Jumbo Frames
(9k) have been enabled and the team is also identified with 9k support (as shown by Get-NetIPInterface).
As the interface is dedicated to migrations and management only (i.e., the corresponding Team is not bound to a Hyper-V Switch, but rather is just a "normal" Team with IP configuration), Hyper-V port does not make a difference here, as there are
no VMs to bind to interfaces on the outbound NIC but just traffic from the Hyper-V base system.
Finally, there are no bandwidth weights and/or QoS rules for the migration traffic bound to the corresponding interface(s).
As I'm able to transfer close to 2GBit/s SMB traffic over the interface (using just a plain file copy), I'm wondering why the SMB(?) transfer of the disk volume during shared-nothing live migration is seemingly limited to somewhere around 700 MBit/s on the
team; looking at the TCP-connections on the remote host, it does seem to use multichannel SMB to copy the disk, but I might be mistaken on that.
Are there any further hints or is there any further information I might offer to diagnose this? I'm currently pretty much stumped on where to go on looking.

Hyper-V Nic Teaming (reserve a nic for host OS)

Whilst setting up nic teaming on my host (server 2012 r2) the OS recommends leaving one nic for host management(access). IS this best practice? Seems like a waste for a nic as the host would hardly ever be accessed after initial setup.
I have 4 nics in total. What is the best practice in this situation?

Depending on if it is a single and the one and only or you build a Cluster you need some networks on your Hyper-V
at least one connection for the Host to do Management.
so in case of a single node with local disks you would create a Team with the 4 Nics and create a Hyper-V Switch with the Option checked for creating that Management OS Adapter what is a so called vNIC on that vSwitch and configure that vNIC with the needed
IP Setting etc...
If you plan a Cluster and also ISCSI/SMB for Storage Access take a look here
http://www.thomasmaurer.ch/2012/07/windows-server-2012-hyper-v-converged-fabric/
You find a few possible ways for teaming and the Switch Settings and also all needed Steps for doing a fully converged Setup via PowerShell.
If you share more Informations on you setup we can give more Details on that.

RAC, Interconnect and CloudControl (a nice to have problem :-) )

Hi,
someone has changed the networkmask for one interconnect interface (eth3). eth3 is up an running, the interconnect ip-address move to the second interconnect interface here eth1. Everything is working fine, and after network is changed back the eth3 interface get his interconnect ip-address and all is fine. But there was no incident in CloudControl no warning that something is going wrong. A user defined metric on GV_$CLUSTER_INTERCONNECTS is no solution, the view shows old values. cluyfy is useful in this case.
best regards
Thomas

Hi,
I think two options
cluvfy and oifcfg to check the network configuration
regards

Question about RAC interconnect

Hi all,
If I want to set up a RAC with multiple interconnects. Is there any specific configuration on the NIC? I am using SUN. Thanks!

Have you checked the RAC installation guides? The biggest thing is to set up each set of interconnects on its own subnet to prevent routing issues.
Ssince I assume you are doing this for redundancy purposes, teamed NICs are another good idea that don't require a lot of re-configuration of Oracle.

Oracle RAC Interconnect, PowerVM VLANs, and the Limit of 20

Hello,
Our company has a requirement to build a multitude of Oracle RAC clusters on AIX using Power VM on 770s and 795 hardware.
We presently have 802.1q trunking configured on our Virtual I/O Servers, and have currently consumed 12 of 20 allowed VLANs for a virtual ethernet adapter. We have read the Oracle RAC FAQ on Oracle Metalink and it seems to otherwise discourage the use of sharing these interconnect VLANs between different clusters. This puts us in a scalability bind; IBM limits VLANs to 20 and Oracle says there is a one-to-one relationship between VLANs and subnets and RAC clusters. We must assume we have a fixed number of network interfaces available and that we absolutely have to leverage virtualized network hardware in order to build these environments. "add more network adapters to VIO" isn't an acceptable solution for us.
Does anyone know if Oracle can afford any flexibility which would allow us to host multiple Oracle RAC interconnects on the same 802.1q trunk VLAN? We will independently guarantee the bandwidth, latency, and redundancy requirements are met for proper Oracle RAC performance, however we don't want a design "flaw" to cause us supportability issues in the future.
We'd like it very much if we could have a bunch of two-node clusters all sharing the same private interconnect. For example:
Cluster 1, node 1: 192.168.16.2 / 255.255.255.0 / VLAN 16
Cluster 1, node 2: 192.168.16.3 / 255.255.255.0 / VLAN 16
Cluster 2, node 1: 192.168.16.4 / 255.255.255.0 / VLAN 16
Cluster 2, node 2: 192.168.16.5 / 255.255.255.0 / VLAN 16
Cluster 3, node 1: 192.168.16.6 / 255.255.255.0 / VLAN 16
Cluster 3, node 2: 192.168.16.7 / 255.255.255.0 / VLAN 16
Cluster 4, node 1: 192.168.16.8 / 255.255.255.0 / VLAN 16
Cluster 4, node 2: 192.168.16.9 / 255.255.255.0 / VLAN 16
etc.
Whereas the concern is that Oracle Corp will only support us if we do this:
Cluster 1, node 1: 192.168.16.2 / 255.255.255.0 / VLAN 16
Cluster 1, node 2: 192.168.16.3 / 255.255.255.0 / VLAN 16
Cluster 2, node 1: 192.168.17.2 / 255.255.255.0 / VLAN 17
Cluster 2, node 2: 192.168.17.3 / 255.255.255.0 / VLAN 17
Cluster 3, node 1: 192.168.18.2 / 255.255.255.0 / VLAN 18
Cluster 3, node 2: 192.168.18.3 / 255.255.255.0 / VLAN 18
Cluster 4, node 1: 192.168.19.2 / 255.255.255.0 / VLAN 19
Cluster 4, node 2: 192.168.19.3 / 255.255.255.0 / VLAN 19
Which eats one VLAN per RAC cluster.

Thank you for your answer!!
I think I roughly understand the argument behind a 2-node RAC and a 3-node or greater RAC. We, unfortunately, were provided with two physical pieces of hardware to virtualize to support production (and two more to support non-production) and as a result we really have no place to host a third RAC node without placing it within the same "failure domain" (I hate that term) as one of the other nodes.
My role is primarily as a system engineer, and, generally speaking, our main goals are eliminating single points of failure. We may be misusing 2-node RACs to eliminate single points of failure since it seems to violate the real intentions behind RAC, which is used more appropriately to scale wide to many nodes. Unfortunately, we've scaled out to only two nodes, and opted to scale these two nodes up, making them huge with many CPUs and lots of memory.
Other options, notably the active-passive failover cluster we have in HACMP or PowerHA on the AIX / IBM Power platform is unattractive as the standby node drives no resources yet must consume CPU and memory resources so that it is prepared for a failover of the primary node. We use HACMP / PowerHA with Oracle and it works nice, however Oracle RAC, even in a two-node configuration, drives load on both nodes unlike with an active-passive clustering technology.
All that aside, I am posing the question to both IBM, our Oracle DBAs (whom will ask Oracle Support). Typically the answers we get vary widely depending on the experience and skill level of the support personnel we get on both the Oracle and IBM sides... so on a suggestion from a colleague (Hi Kevin!) I posted here. I'm concerned that the answer from Oracle Support will unthinkingly be "you can't do that, my script says to tell you the absolute most rigid interpretation of the support document" while all the time the same document talks of the use of NFS and/or iSCSI storage eye roll
We have a massive deployment of Oracle EBS and honestly the interconnect doesn't even touch 100mbit speeds even though the configuration has been checked multiple times by Oracle and IBM and with the knowledge that Oracle EBS is supposed to heavily leverage RAC. I haven't met a single person who doesn't look at our environment and suggest jumbo frames. It's a joke at this point... comments like "OMG YOU DON'T HAVE JUMBO FRAMES" and/or "OMG YOU'RE NOT USING INFINIBAND WHATTA NOOB" are commonplace when new DBAs are hired. I maintain that the utilization numbers don't support this.
I can tell you that we have 8Gb fiber channel storage and 10Gb network connectivity. I would probably assume that there were a bottleneck in the storage infrastructure first. But alas, I digress.
Mainly I'm looking for a real-world answer to this question. Aside from violating every last recommendation and making oracle support folk gently weep at the suggestion, are there any issues with sharing interconnects between RAC environments that will prevent it's functionality and/or reduce it's stability?
We have rapid spanning tree configured, as far as I know, and our network folks have tuned the timers razor thin. We have Nexus 5k and Nexus 7k network infrastructure. The typical issues you'd fine with standard spanning tree really don't affect us because our network people are just that damn good.

Network adapter for the private interface for RAC

Hi guys
This is the specification for the Network adapter for the private interface in RAC:
The network adapter for the private interface must support the user datagram protocol (UDP) using high-speed network adapters and a network switch that supports TCP/IP (Gigabit Ethernet or better).
Do you have a document where I can have more deep details about it ?
Thanks

user2931261 wrote:
This is the specification for the Network adapter for the private interface in RAC:
The network adapter for the private interface must support the user datagram protocol (UDP) using high-speed network adapters and a network switch that supports TCP/IP (Gigabit Ethernet or better).Note that TCP is also used in addition to UDP over the Interconnect. As for the statement that the NIC must support UDP - wrong statement to make. The NIC does not support UDP. The IP (Internet Protocol) stack does. The NIC supports the bottom layer of the ISO model. UDP is at layer 4.
As for your Interconnect, you have two basic choices. Gigabyte Ethernet or Infiniband.
For GigE you need of course a GigE NIC and a GigE switch, and a cable to wire the NIC into the switch.
For Infiniband you need a HCA card (e.g. Mellanox InfiniHost PCI card) and an Infiniband switch. As HCA cards are dual port, it make sense to get 2 cables and wire both ports into the switch and bond those 2 ports as a single logical NIC. You can run IPoIB (IP over Infiniband), which means that the 2 ports on the HCA will be seen by the o/s as NICs - and bonding is supported which enables you to create a single logical NIC on top of these 2 NICs and thus have full redundancy.
Also note that GigE typically is only 1Gb/s. 10 Gb/s is available, but expensive (especially the switch). Infiniband typically is QDR rate today that provides you with 40Gb/s pipes. So each port on the HCA will be 40Gb and the bonded port will thus be 2x40Gb/s.
Infiniband is a lot faster and more scalable for an Interconnect than GigE.
Also, because of the capacity and flexibility and low latency of Infiniband, you can also use it for your storage fabric layer (running your storage protocol over it).
Typically you will need to get HBA cards (dual fibre port) to wire the RAC node to the storage array's fibre channel switch. These HBA ports are 2Gb/s.
You can eliminate these HBAs and fibre cables by using a storage protocol like SRP (Scsi RDMA Protocol - RDMA=Remote Direct Memory Access) - running SRP over your Infiniband infrastructure, together with your Interconnect.
This is btw what Oracle's Database Machine and Exadata Storage servers use (and one the reasons they have broken all RAC performance records).

RAC - Interconnect traffic

In the client place the architecture team wants to implement a a node RAC cluster on Sun Solaris on Oracle 10g. But in order to minimize the interconnect traffic they want applications to connect only to one node and the other node will provide fail over capability.
Though this can be achieved with less complexity thru physical standby, theoretically if in a 2 node RAC cluster if we just use one node will it eliminate interconnect traffic?
I guess that there will not be any requests from the second node for blocks held by the first node so no transfer of blocks over interconnect will take place. But still the second node has to know about the blocks held by the first node in order to recover in case the first node crashes(cache fusion?). Right?
TIA
RadKrish

just my 0.02 cents on this .. Why would you want to invest in a High Availability Solution ( RAC ) and attempt to achieve Disaster Recovery ( Failover ) through it ..
Why would you want to minimize the Interconnect traffic in the first place .. A decent Switch plus Gigabit Ethernet / Cat5 cables is all you need for the Interconnect and its known to work well with most types of RAC Setup's ..
With Dynamic Resource Remastering , mastership of blocks across the active nodes get distributed based on access patterns .. so with your kind of setup , almost all the blocks would be mastered on Node:1 after a period of sustained activity .. Further Instance recovery happens using information from the Redo Log Thread of the crashed Node so it should not affect Interconnect traffic ..
Vishwa

Ethernet Cards(NICs) in RAC setup

DB version: 11g
I came across this nice link about RAC setup
http://www.akadia.com/services/ora_rac.html
Question 1.
In the diagram, I can see eth0 and eth1 in both nodes. They are physical ethernet cards (NICs). Right?
Question 2.
Each node having 2 ethernet cards is industry's standard practice. Right?
Question 3.
For Private InterConnect ===>One ethernet card from each node, connected via Switch1
For Public InterConnect plus Connection to Shared storage ====> One ethernet card from each node, connected via Switch2Is this the standard practice in Production DBs?

resistanceIsFruitful wrote:
In the diagram, I can see eth0 and eth1 in both nodes. They are physical ethernet cards (NICs). Right? Yes. (or virtual ones in a virtualised environment).
Question 2.
Each node having 2 ethernet cards is industry's standard practice. Right?Yes and No. Yes, typically 2 ethernet ports. Servers usually will have 4 or more ports, if one also includes the management ethernet ports of the server. (but these cannot be used by o/s or RAC/Grid software).
If one wants to have failover for either the private or public interfaces, then at least 2 physical interfaces are needed for each. E.g. eth0 and eth1 will be bonded as logical interface bond0 for the public network, and eth2 and eth3 bonded as bond1 for the private network.
Question 3.
For Private InterConnect ===>One ethernet card from each node, connected via Switch1
For Public InterConnect plus Connection to Shared storage ====> One ethernet card from each node, connected via Switch2
Is this the standard practice in Production DBs?Yes - fundamentally this is correct. You want your Interconnect to be a fully private network with a dedicated switch. That said, high availability does not mean all your connectivity eggs in a single basket. If you have a single private switch, that is a single point of failure for the entire cluster. If you have a single private interface, that is a single point of failure for that node to the Interconnect (cable failures are not uncommon).
Thus you should be looking at redudancy at both the switch and the NIC/interface levels.

Teamed NICs for RAC interconnect

Similar Messages

Maybe you are looking for