NICs for Private Interconnect redundancy

DB/Grid version : 11.2.0.2
Platform : AIX 6.1
We are going to install a 2-node RAC on AIX (that thing which is almost good as Solaris )
Our primary private interconnect is
### Primary Private Interconnect
169.21.204.1      scnuprd186-privt1.mvtrs.net scnuprd186-privt1
169.21.204.4      scnuprd187-privt1.mvtrs.net scnuprd187-privt1For Cluster inteconnect's redundancy , Unix team has attached an extra NIC for each node with an extra Gigabit-ethernet switch for these NICs.
###Redundant Private Interconnect attached to the server
169.21.204.2      scnuprd186-privt2.mvtrs.net scnuprd186-privt2 # Node1's newly attached redundant NIC
169.21.204.5      scnuprd187-privt2.mvtrs.net scnuprd187-privt2 # Node2's newly attached redundant NICExample borrowed from citizen2's post
Apparently I have 2 ways to implement cluster inteconnect's redundancy
Option1. NIC bonding at OS level
Option2. Let grid software do it
Question1. Which is better : Option 1 or 2 ?
Question2.
Regarding Option2.
From googling and OTN , i gather that , during grid installation you just provide 169.21.204.0 for cluster inteconnect and grid will identify the redundant NIC and switch. And if something goes wrong with the Primary Interconnect setup (shown above) , grid will automatically re-route interconnect traffic using the redundant NIC setup. Is this correct ?
Question 3.
My colleague tells me , for the redundant Switch (Gigabit) Unless I configure some Multicasting (AIX specific), I could get errors during installation. He doesn't clearly what it was ? Anyone faced Multicasting related issue on this ?

Hi,
My recommendation is to you use the AIX EtherChannel.
The EtherCannel of AIX is much more powerfull and stable compared with HAIP.
See how setup AIX EtherChannel on 10 Gigabit Ethernet interfaces
http://levipereira.wordpress.com/2011/01/26/setting-up-ibm-power-systems-10-gigabit-ethernet-ports-and-aix-6-1-etherchannel-for-oracle-rac-private-interconnectivity/
If you choose use HAIP I recommend you read this note, and find all notes about bugs of HAIP on AIX.
11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip [ID 1210883.1]
ASM Crashes as HAIP Does not Failover When Two or More Private Network Fails [ID 1323995.1]
About Multicasting read it:
Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1]
Regards,
Levi Pereira

Similar Messages

Private Interconnect redundancy

Grid Version : 11.2.0.2
OS : Solaris 10 on HP Proliant
Currently we have a 2-node RAC running with 4 live DBs.
Currently our private interconnect is
### Current Private Interconnect
169.21.204.1      scnuprd186-privt1.mvtrs.net scnuprd186-privt1
169.21.204.4      scnuprd187-privt1.mvtrs.net scnuprd187-privt1To have redundancy for private interconnect , After repeated requests, our Unix team has finally attached a redundant NIC for each node with a redundant Gigabit-ethernet switch.
So, we need to add the below NIC to the CRS. How can we do that?
###Redundant Private Interconnect (currently attached to the server, but yet to be 'included' in the cluster)
169.21.204.2      scnuprd186-privt2.mvtrs.net scnuprd186-privt2 # Node1's newly attached redundant NIC
169.21.204.5      scnuprd187-privt2.mvtrs.net scnuprd187-privt2 # Node2's newly attached redundant NIC

Citizen_2 wrote:
Grid Version : 11.2.0.2
OS : Solaris 10 on HP Proliant
Currently we have a 2-node RAC running with 4 live DBs.
Currently our private interconnect is
### Current Private Interconnect
169.21.204.1      scnuprd186-privt1.mvtrs.net scnuprd186-privt1
169.21.204.4      scnuprd187-privt1.mvtrs.net scnuprd187-privt1To have redundancy for private interconnect , After repeated requests, our Unix team has finally attached a redundant NIC for each node with a redundant Gigabit-ethernet switch.You can use IPMP (IP MultiPath) in Solaris.
First, note that these should be NON-ROUTABLE addresses configured on a PRIVATE-Dedicated Switch. It would look something like this:
169.21.204.1 scnuprd186-privt1-IPMPvip.mvtrs.net scnuprd186-privt1-IPMPvip
169.21.204.2 scnuprd186-privt1-nic1.mvtrs.net scnuprd186-privt1-nic1 eth2
169.21.204.3 scnuprd186-privt1-nic2.mvtrs.net scnuprd186-privt1-nic2 eth3
169.21.204.4 scnuprd187-privt1-IPMPvip.mvtrs.net scnuprd187-privt1-IPMPvip
169.21.204.5 scnuprd187-privt1-nic1.mvtrs.net scnuprd187-privt1-nic1 eth2
169.21.204.6 scnuprd187-privt1-nic2.mvtrs.net scnuprd187-privt1-nic2 eth3
IPMP has a "real address" for each "real" interface and the IPMPvip's will "float" between the eth2 and eth3 devices depending on which one is active. Similar to the way the host vip can "float" between nodes. It is the IPMPvip addresses that are provided to the CRS configuration.
I have used this on Sun 6900's and it worked great.
Now, it can get extremely complicated if you were to also use IPMP on the public interfaces as well. It does work, you just need to pay attention to how you configure it.
>
So, we need to add the below NIC to the CRS. How can we do that?
###Redundant Private Interconnect (currently attached to the server, but yet to be 'included' in the cluster)
169.21.204.2      scnuprd186-privt2.mvtrs.net scnuprd186-privt2 # Node1's newly attached redundant NIC
169.21.204.5      scnuprd187-privt2.mvtrs.net scnuprd187-privt2 # Node2's newly attached redundant NIC

RAC Private interconnect redundancy

Hello All,
We are designing (implementation will be done later) a 2-node RAC Database with GI having version 12.1.0.2 and RDBMS S/W having version 11.2.0.4.
We want to make private interconnect redundant but sysadmin does not have two same bandwidth channels, he is giving two NICs with 10Gbe (Giga bit Ethernet) and 1Gbe respectively.
I got to know that 1 Gbe is sufficient for GES and GCS but will this architecture work fine means any harm in having 2 different bandwidth channels also in case of failure of 10Gbe interface definitely there will be performance degradation.
Thanks,
Hemant.

DO NOT use two different network bandwidths for your Cluster Interconnect. With two physical NICs, you will either resort to NIC bonding or HAIP, the latter being the recommendation from Oracle Corp since you are using 12c. In either case, both NIC's will be used equally. This means some traffic on the private network will be 'slower' than the other traffic. You do run the risk of having performance issues with this configuration.
Also...there are two reasons for implementing multiple NICs for the Cluster Interconnect, performance and high availability. I've addressed performance above. On the HA side, dual NICs mean that if one channel goes down, the other channel is available and the cluster can stay operational. There is a law of the universe that says if you have 10gE on one side and 1gE on the other side, you have a 99% chance that if one channel goes down, it will be the 10gE one. Which means you may not have enough bandwidth on the remaining channel.
Cheers,
Brian

Teamed NICs for RAC interconnect

Hi there,
We have an Oralce 10g RAC with 2 nodes. there are only one NIC for RAC interconnect in both servers.
now we want to add one redundant NIC into each server for RAC interconnect as well.
Could you please guide me some documents about this "teamed NICs for RAC interconnect "?
Your help is greatly appreciated!
Thanks,
Scott

Search around for NIC bonding. The exact process will depend on your OS.
Linux, see Metalink note 298891.1 - Configuring Linux for the Oracle 10g VIP or private interconnect using bonding driver
Regards,
Greg Rahn
http://structureddata.org

Node crashes when enabling RDS for private interconnect.

OS: oel6.3 - 2.6.39-300.17.2.el6uek.x86_64
Grid and DB: 11.2.0.3.4
This is a two node Standard Edition cluster.
The node crashes upon restart of clusterware after following the instructions from note:751343.1 (RAC Support for RDS Over Infiniband) to enable RDS.
The cluster is running fine using ipoib for the cluster_interconnect.
1) As the ORACLE_HOME/GI_HOME owner, stop all resources (database, listener, ASM etc) that's running from the home. When stopping database, use NORMAL or IMMEDIATE option.
2) As root, if relinking 11gR2 Grid Infrastructure (GI) home, unlock GI home: GI_HOME/crs/install/rootcrs.pl -unlock
3) As the ORACLE_HOME/GI_HOME owner, go to ORACLE_HOME/GI_HOME and cd to rdbms/lib
4) As the ORACLE_HOME/GI_HOME owner, issue "make -f ins_rdbms.mk ipc_rds ioracle"
5) As root, if relinking 11gR2 Grid Infrastructure (GI) home, lock GI home: GI_HOME/crs/install/rootcrs.pl -patch
Looks to abend when asm tries to start with the message below on the console.
I have a service request open for this issue but, I am hoping someone may have seen this and has
some way around it.
Thanks
Alan
kernel BUG at net/rds/ib_send.c:547!
invalid opcode: 0000 [#1] SMP
CPU 2
Modules linked in: 8021q garp stp llc iptable_filter ip_tables nfs lockd
fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand powernow_k8
freq_table mperf rds_rdma rds_tcp rds ib_ipoib rdma_ucm ib_ucm ib_uverbs
ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa sr_mod cdrom microcode
serio_raw pcspkr ghes hed k10temp hwmon amd64_edac_mod edac_core
edac_mce_amd i2c_piix4 i2c_core sg igb dca mlx4_ib ib_mad ib_core
mlx4_en mlx4_core ext4 mbcache jbd2 usb_storage sd_mod crc_t10dif ahci
libahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded:
scsi_wait_scan]
Pid: 4140, comm: kworker/u:1 Not tainted 2.6.39-300.17.2.el6uek.x86_64
#1 Supermicro BHDGT/BHDGT
RIP: 0010:[<ffffffffa02db829>] [<ffffffffa02db829>]
rds_ib_xmit+0xa69/0xaf0 [rds_rdma]
RSP: 0018:ffff880fb84a3c50 EFLAGS: 00010202
RAX: ffff880fbb694000 RBX: ffff880fb3e4e600 RCX: 0000000000000000
RDX: 0000000000000030 RSI: ffff880fbb6c3a00 RDI: ffff880fb058a048
RBP: ffff880fb84a3d30 R08: 0000000000000fd0 R09: ffff880fbb6c3b90
R10: 0000000000000000 R11: 000000000000001a R12: ffff880fbb6c3a00
R13: ffff880fbb6c3a00 R14: 0000000000000000 R15: ffff880fb84a3d90
FS: 00007fd0a3a56700(0000) GS:ffff88101e240000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000002158ca2 CR3: 0000000001783000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/u:1 (pid: 4140, threadinfo ffff880fb84a2000, task
ffff880fae970180)
Stack:
0000000000012200 0000000000012200 ffff880f00000000 0000000000000000
000000000000e5b0 ffffffff8115af81 ffffffff81b8d6c0 ffffffffa02b2e12
00000001bf272240 ffffffff81267020 ffff880fbb6c3a00 0000003000000002
Call Trace:
[<ffffffff8115af81>] ? __kmalloc+0x1f1/0x200
[<ffffffffa02b2e12>] ? rds_message_alloc+0x22/0x90 [rds]
[<ffffffff81267020>] ? sg_init_table+0x30/0x50
[<ffffffffa02b2db2>] ? rds_message_alloc_sgs+0x62/0xa0 [rds]
[<ffffffffa02b31e4>] ? rds_message_map_pages+0xa4/0x110 [rds]
[<ffffffffa02b4f3b>] rds_send_xmit+0x38b/0x6e0 [rds]
[<ffffffff81089d53>] ? cwq_activate_first_delayed+0x53/0x100
[<ffffffffa02b6040>] ? rds_recv_worker+0xc0/0xc0 [rds]
[<ffffffffa02b6075>] rds_send_worker+0x35/0xc0 [rds]
[<ffffffff81089fd6>] process_one_work+0x136/0x450
[<ffffffff8108bbe0>] worker_thread+0x170/0x3c0
[<ffffffff8108ba70>] ? manage_workers+0x120/0x120
[<ffffffff810907e6>] kthread+0x96/0xa0
[<ffffffff81515544>] kernel_thread_helper+0x4/0x10
[<ffffffff81090750>] ? kthread_worker_fn+0x1a0/0x1a0
[<ffffffff81515540>] ? gs_change+0x13/0x13
Code: ff ff e9 b1 fe ff ff 48 8b 0d b4 54 4b e1 48 89 8d 70 ff ff ff e9
71 ff ff ff 83 bd 7c ff ff ff 00 0f 84 f4 f5 ff ff 0f 0b eb fe <0f> 0b
eb fe 44 8b 8d 48 ff ff ff 41 b7 01 e9 51 f6 ff ff 0f 0b
RIP [<ffffffffa02db829>] rds_ib_xmit+0xa69/0xaf0 [rds_rdma]
RSP <ffff880fb84a3c50>
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.39-300.17.2.el6uek.x86_64
([email protected]) (gcc version 4.4.6 20110731 (Red
Hat 4.4.6-3) (GCC) ) #1 SMP Wed Nov 7 17:48:36 PST 2012
Command line: ro root=UUID=5ad1a268-b813-40da-bb76-d04895215677
rd_DM_UUID=ddf1_stor rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD
SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us numa=off
console=ttyS1,115200n8 irqpoll maxcpus=1 nr_cpus=1 reset_devices
cgroup_disable=memory mce=off memmap=exactmap memmap=538K@64K
memmap=130508K@770048K elfcorehdr=900556K memmap=72K#3668608K
memmap=184K#3668680K
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000100 - 0000000000096800 (usable)
BIOS-e820: 0000000000096800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000dfe90000 (usable)
BIOS-e820: 00000000dfe9e000 - 00000000dfea0000 (reserved)
BIOS-e820: 00000000dfea0000 - 00000000dfeb2000 (ACPI data)
BIOS-e820: 00000000dfeb2000 - 00000000dfee0000 (ACPI NVS)
BIOS-e820: 00000000dfee0000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000ffe00000 - 0000000100000000 (reserved)

I believe OFED version is 1.5.3.3 but I am not sure if this is correct.
We have not added any third parry drivers. All that has been done to add infiniband to our build is
a yum groupinstall iInfiniband support.
I have not tries rds-stress but rds-ping works fine and rds-info seems fine.
A service request has been opened but so far I have had better response here.
oracle@blade1-6:~> rds-info
RDS IB Connections:
LocalAddr RemoteAddr LocalDev RemoteDev
10.10.0.116 10.10.0.119 fe80::25:90ff:ff07:df1d fe80::25:90ff:ff07:e0e5
TCP Connections:
LocalAddr LPort RemoteAddr RPort HdrRemain DataRemain SentNxt ExpectUna SeenUna
Counters:
CounterName Value
conn_reset 5
recv_drop_bad_checksum 0
recv_drop_old_seq 0
recv_drop_no_sock 1
recv_drop_dead_sock 0
recv_deliver_raced 0
recv_delivered 18
recv_queued 18
recv_immediate_retry 0
recv_delayed_retry 0
recv_ack_required 4
recv_rdma_bytes 0
recv_ping 14
send_queue_empty 18
send_queue_full 0
send_lock_contention 0
send_lock_queue_raced 0
send_immediate_retry 0
send_delayed_retry 0
send_drop_acked 0
send_ack_required 3
send_queued 32
send_rdma 0
send_rdma_bytes 0
send_pong 14
page_remainder_hit 0
page_remainder_miss 0
copy_to_user 0
copy_from_user 0
cong_update_queued 0
cong_update_received 1
cong_send_error 0
cong_send_blocked 0
ib_connect_raced 4
ib_listen_closed_stale 0
ib_tx_cq_call 6
ib_tx_cq_event 6
ib_tx_ring_full 0
ib_tx_throttle 0
ib_tx_sg_mapping_failure 0
ib_tx_stalled 16
ib_tx_credit_updates 0
ib_rx_cq_call 33
ib_rx_cq_event 38
ib_rx_ring_empty 0
ib_rx_refill_from_cq 0
ib_rx_refill_from_thread 0
ib_rx_alloc_limit 0
ib_rx_credit_updates 0
ib_ack_sent 4
ib_ack_send_failure 0
ib_ack_send_delayed 0
ib_ack_send_piggybacked 0
ib_ack_received 3
ib_rdma_mr_alloc 0
ib_rdma_mr_free 0
ib_rdma_mr_used 0
ib_rdma_mr_pool_flush 8
ib_rdma_mr_pool_wait 0
ib_rdma_mr_pool_depleted 0
ib_atomic_cswp 0
ib_atomic_fadd 0
iw_connect_raced 0
iw_listen_closed_stale 0
iw_tx_cq_call 0
iw_tx_cq_event 0
iw_tx_ring_full 0
iw_tx_throttle 0
iw_tx_sg_mapping_failure 0
iw_tx_stalled 0
iw_tx_credit_updates 0
iw_rx_cq_call 0
iw_rx_cq_event 0
iw_rx_ring_empty 0
iw_rx_refill_from_cq 0
iw_rx_refill_from_thread 0
iw_rx_alloc_limit 0
iw_rx_credit_updates 0
iw_ack_sent 0
iw_ack_send_failure 0
iw_ack_send_delayed 0
iw_ack_send_piggybacked 0
iw_ack_received 0
iw_rdma_mr_alloc 0
iw_rdma_mr_free 0
iw_rdma_mr_used 0
iw_rdma_mr_pool_flush 0
iw_rdma_mr_pool_wait 0
iw_rdma_mr_pool_depleted 0
tcp_data_ready_calls 0
tcp_write_space_calls 0
tcp_sndbuf_full 0
tcp_connect_raced 0
tcp_listen_closed_stale 0
RDS Sockets:
BoundAddr BPort ConnAddr CPort SndBuf RcvBuf Inode
0.0.0.0 0 0.0.0.0 0 131072 131072 340441
RDS Connections:
LocalAddr RemoteAddr NextTX NextRX Flg
10.10.0.116 10.10.0.119 33 38 --C
Receive Message Queue:
LocalAddr LPort RemoteAddr RPort Seq Bytes
Send Message Queue:
LocalAddr LPort RemoteAddr RPort Seq Bytes
Retransmit Message Queue:
LocalAddr LPort RemoteAddr RPort Seq Bytes
10.10.0.116 0 10.10.0.119 40549 32 0
oracle@blade1-6:~> cat /etc/rdma/rdma.conf
# Load IPoIB
IPOIB_LOAD=yes
# Load SRP module
SRP_LOAD=no
# Load iSER module
ISER_LOAD=no
# Load RDS network protocol
RDS_LOAD=yes
# Should we modify the system mtrr registers? We may need to do this if you
# get messages from the ib_ipath driver saying that it couldn't enable
# write combining for the PIO buffs on the card.
# Note: recent kernels should do this for us, but in case they don't, we'll
# leave this option
FIXUP_MTRR_REGS=no
# Should we enable the NFSoRDMA service?
NFSoRDMA_LOAD=yes
NFSoRDMA_PORT=2050
oracle@blade1-6:~> /etc/init.d/rdma status
Low level hardware support loaded:
     mlx4_ib
Upper layer protocol modules:
     rds_rdma ib_ipoib
User space access modules:
     rdma_ucm ib_ucm ib_uverbs ib_umad
Connection management modules:
     rdma_cm ib_cm iw_cm
Configured IPoIB interfaces: none
Currently active IPoIB interfaces: ib0

Redundancy at Private interconnect.

Hi,
We are planning to setup a 2 node RAC. Our system admin has provided 2 nics for private interconnect. We were looking to use both as private interconnect.
Operating environment
Solaris 10
Oracle 10g R2 (clusterware, rdbms)
Current configuration of NICs provided for interconnect.
nxge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 6
inet 192.168.1.119 netmask ffffff00 broadcast 192.168.1.255
ether 0:21:28:69:a7:37
nxge2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 7
inet 192.168.2.119 netmask ffffff00 broadcast 192.168.2.255
ether 0:21:28:69:a7:38
My questions:
As per oracle support note "How to Setup IPMP as Cluster Interconnect (Doc ID 368464.1)"
we can use IPMP grouping for Interconnect, but it is not very clear.
1) If I use IPMP group do i need to specify only one physical ip as cluster_interconnect or all the ips associated to NIC. (Will this allow load balancing or only failover).
2) If we do not want to use IPMP can I specify all the IP address of NICs in cluster_interconnects parameter (This will not allow failover only load balancing).
Regards
Veera

user7636989 wrote:
Hi,
We are planning to setup a 2 node RAC. Our system admin has provided 2 nics for private interconnect. We were looking to use both as private interconnect.
Operating environment
Solaris 10
Oracle 10g R2 (clusterware, rdbms)
Current configuration of NICs provided for interconnect.
nxge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 6
inet 192.168.1.119 netmask ffffff00 broadcast 192.168.1.255
ether 0:21:28:69:a7:37
nxge2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 7
inet 192.168.2.119 netmask ffffff00 broadcast 192.168.2.255
ether 0:21:28:69:a7:38A prerequsite for IPMP is that the participating IP interfaces should be in the same IP broadcast
subnet, but your output above shows them to be on different subnets (192.168.1.0/24 and
192.168.2.0/24). That would need to be fixed before you can use IPMP (which supports
both failover and loadbalancing). And if you used IPMP, then you would need to set up
the cluster_interconnect to be the data address(es) (the one(s) that is(are) not set up with
the NOFAILOVER flag).
>
My questions:
As per oracle support note "How to Setup IPMP as Cluster Interconnect (Doc ID 368464.1)"
we can use IPMP grouping for Interconnect, but it is not very clear.
1) If I use IPMP group do i need to specify only one physical ip as cluster_interconnect or all the ips associated to NIC. (Will this allow load balancing or only failover).
2) If we do not want to use IPMP can I specify all the IP address of NICs in cluster_interconnects parameter (This will not allow failover only load balancing).
Regards
Veera

Copper cable / GigE Copper Interface as Private Interconnect for Oracle RAC

Hello Gurus
Can some one confirm if the copper Cables ( Cat5/RJ45) can be used for Gig Ethernet i.e. Private interconnects for deploying Oracle RAC 9.x or 10gR2 on Solaris 9/10 .
i am planning to use 2 X GigE Interfaces (one port each from X4445 Quad Port Ethernet Adapters) & Planning to connect it using copper cables ( all the documents that i came across is been refering to the fiber cables for Private Interconnects , connecting GigE Interfaces , so i am getting bit confused )
would appretiate if some one can throw some lights on the same.
regards,
Nilesh Naik
thanks

Cat5/RJ45 can be used for Gig Ethernet Private interconnects for Oracle RAC. I would recommend trunking the two or more interconnects for redundancy. The X4445 adapters are compatible with the Sun Trunking 1.3 software (http://www.sun.com/products/networking/ethernet/suntrunking/). If you have servers that support the Nemo framework (bge, e1000g, xge, nge, rge, ixgb), you can use the Solaris 10 trunking software, dladmin.
We have a couple of SUN T2000 servers and are using the onboard GigE ports for the Oracle 10gR2 RAC interconnects. We upgraded the onboard NIC drivers to the e1000g and used the Solaris 10 trunking software. The next update of Solaris will have the e1000g drivers as the default for the SUN T2000 servers.

Gig Ethernet V/S SCI as Cluster Private Interconnect for Oracle RAC

Hello Gurus
Can any one pls confirm if it's possible to configure 2 or more Gigabit Ethernet interconnects ( Sun Cluster 3.1 Private Interconnects) on a E6900 cluster ?
It's for a High Availability requirement of Oracle 9i RAC. i need to know ,
1) can i use gigabit ethernet as Private cluster interconnect for Deploying Oracle RAC on E6900 ?
2) What is the recommended Private Cluster Interconnect for Oracle RAC ? GiG ethernet or SCI with RSM ?
3) How about the scenarios where one can have say 3 X Gig Ethernet V/S 2 X SCI , as their cluster's Private Interconnects ?
4) How the Interconnect traffic gets distributed amongest the multiple GigaBit ethernet Interconnects ( For oracle RAC) , & is anything required to be done at oracle Rac Level to enable Oracle to recognise that there are multiple interconnect cards it needs to start utilizing all of the GigaBit ethernet Interfaces for transfering packets ?
5) what would happen to Oracle RAC if one of the Gigabit ethernet private interconnects fails
Have tried searching for this info but could not locate any doc that can precisely clarify these doubts that i have .........
thanks for the patience
Regards,
Nilesh

Answers inline...
Tim
Can any one pls confirm if it's possible to configure
2 or more Gigabit Ethernet interconnects ( Sun
Cluster 3.1 Private Interconnects) on a E6900
cluster ?Yes, absolutely. You can configure up to 6 NICs for the private networks. Traffic is automatically striped across them if you specify clprivnet0 to Oracle RAC (9i or 10g). That is TCP connections and UDP messages.
It's for a High Availability requirement of Oracle
9i RAC. i need to know ,
1) can i use gigabit ethernet as Private cluster
interconnect for Deploying Oracle RAC on E6900 ? Yes, definitely.
2) What is the recommended Private Cluster
Interconnect for Oracle RAC ? GiG ethernet or SCI
with RSM ? SCI is or is in the process of being EOL'ed. Gigabit is usually sufficient. Longer term you may want to consider Infiniband or 10 Gigabit ethernet with RDS.
3) How about the scenarios where one can have say 3 X
Gig Ethernet V/S 2 X SCI , as their cluster's
Private Interconnects ? I would still go for 3 x GbE because it is usually cheaper and will probably work just as well. The latency and bandwidth differences are often masked by the performance of the software higher up the stack. In short, unless you tuned the heck out of your application and just about everything else, don't worry too much about the difference between GbE and SCI.
4) How the Interconnect traffic gets distributed
amongest the multiple GigaBit ethernet Interconnects
( For oracle RAC) , & is anything required to be done
at oracle Rac Level to enable Oracle to recognise
that there are multiple interconnect cards it needs
to start utilizing all of the GigaBit ethernet
Interfaces for transfering packets ?You don't need to do anything at the Oracle level. That's the beauty of using Oracle RAC with Sun Cluster as opposed to RAC on its own. The striping takes place automatically and transparently behind the scenes.
5) what would happen to Oracle RAC if one of the
Gigabit ethernet private interconnects fails It's completely transparent. Oracle will never see the failure.
Have tried searching for this info but could not
locate any doc that can precisely clarify these
doubts that i have .........This is all covered in a paper that I have just completed and should be published after Christmas. Unfortunately, I cannot give out the paper yet.
thanks for the patience
Regards,
Nilesh

What is acceptable level of Private Interconnect Latency for RAC

We have build 3 Node RAC on RHEL5.4 on VMware.
There is node eviction problem due to loss of Network Heartbeat.
ocssd.log:[ CSSD]2010-03-05 17:48:21.908 [84704144] >TRACE: clssnmReadDskHeartbeat: node 3, vm-lnx-rds1173, has a disk HB, but no network HB, DHB has rcfg 0, wrtcnt, 2, LATS 1185024, lastSeqNo 2, timestamp 1267791501/1961474
Ping statistics from Node2 to Node1 are as below
--- rds1171-priv ping statistics ---
443 packets transmitted, 443 received, 0% packet loss, time 538119ms
rtt min/avg/max/mdev = 0.150/2.030/630.212/29.929 ms
[root@vm-lnx-rds1172 oracle]#
Can this be reason for Node eviction? What is acceptable level of of private interconnect latency for RAC ?

What is acceptable level of of private interconnect latency for RAC ?Normal local network latency should be enough. By the way latency settings are very generous.
Can you check if your to-be-evicted node runs and is reachable when seeing the node eviction messages?
In addition to that: Can you check the log files of the eviced node. Check for time stamps around "2010-03-05 17:48:21.908". Make sure all systems are NTP synchronized.
Ronny Egner
My Blog: http://blog.ronnyegner-consulting.de

Private Interconnect: Should any nodes other than RAC nodes have one?

The contractors that set up our four-node production 10g RAC (and a standalone development server) also assigned private interconnect addresses to 2 Apache/ApEx servers and a standalone development database server.
There are service names in the tnsnames.ora on all servers in our infrastructure referencing these private interconnects- even the non-rac member servers. The nics on these servers are not bound for failover with the nics bound to the public/VIP addresses. These nics are isolated on their own switch.
Could this configuration be related to lost heartbeats or voting disk errors? We experience rac node expulsions and even arbitrary bounces (reboots!) of all the rac nodes.

I do not have access to the contractors. . . .can only look at what they have left behind and try to figure out their intention. . .
I am reading the Ault/Tumha book Oracle 10g Grid and Real Application Clusters and looking through our own settings and config files and learning srvctl and crsctl commands from their examples. Also googling and OTN searching through the library full of documentation. . .
I still have yet to figure out if the private interconnect spoken about so frequently in cluster configuration documents are the binding to the set of node.vip address specifications in the tnsnames.ora (bound the the first eth adaptor along with the public ip addresses for the nodes) or the binding on the second eth adaptor to the node.prv addresses not found in the local pfile, in the tnsnames.ora, or the listener.ora (but found at the operating system level in the ifconfig). If the node.prv addresses are not the private interconnect then can anyone tell me that they are for?

Mysterious IP associated with Private Interconnect appears in ifconfig output

Grid Version: 11.2.0.4
Platform : Oracle Linux 6.4
We use bond1 for our Private Interconnect. The subnet for this is 10.5.51.xxxx
But another IP appears in ifconfig and oifcfg iflist output. This is shown under bond1 in ifconfig . I have shown it in red below.
This is seems to be configured by Clusterware. Anyone one knows what this is and the role it plays ?
# /u01/grid/product/11.2/bin/oifcfg iflist
bond0 10.5.19.0
bond1 10.5.51.0
bond1 169.254.0.0
bond2 10.5.12.0
bond3 10.5.34.0
# /u01/grid/product/11.2/bin/oifcfg getif
bond1 10.5.51.0 global cluster_interconnect
bond2 10.5.12.0 global public
# ifconfig -a
bond0     Link encap:Ethernet HWaddr A0:36:9F:26:2A:28
          inet addr:10.5.19.46 Bcast:10.5.19.255 Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
          RX packets:21093181 errors:0 dropped:4246520 overruns:0 frame:0
          TX packets:8781028 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:15996129115 (14.8 GiB) TX bytes:10913945403 (10.1 GiB)
bond1     Link encap:Ethernet HWaddr A0:36:9F:26:2A:29
          inet addr:10.5.51.25 Bcast:10.5.51.255 Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1
          RX packets:16957228829 errors:0 dropped:91363134 overruns:91350881 frame:0
          TX packets:16507123680 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:22253066948104 (20.2 TiB) TX bytes:18486193782633 (16.8 TiB)
bond1:1   Link encap:Ethernet HWaddr A0:36:9F:26:2A:29
          inet addr:169.254.123.209 Bcast:169.254.255.255 Mask:255.255.0.0
          UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1
<output snipped>

This is expected. From release 11.2.0.2 (if I remember correctly) your interconnect traffic goes over a 169.x.x.x network, configured by HAIP. You are meant to expose all the NICs to Oarcle and let HAIP handle load balancing and failure: do not use bonding.
I described the mechanism in a tutorial I recorded some time ago, I think it was this one:
http://skillbuilders.com/webinars/webinar.cfm?id=79&title=Oracle%2011g%20/%2012c%20Grid%20Infrastructure%20Free%20Tutorial
John Watson
Oracle Certified Master DBA

Oracle RAC and crossover cable for private network

Hi,
I have the following configuration: two database servers, each has four network cards, two for public network and two for private, cluster network. Each public card has own IP-address and both have virtual IP-address, defined in operation system (SLES-9) for redundancy. Because I have only two machines in the cluster I want to connect the two machines for private with crossover cable without switch. For redundancy I want to make two connections between machines. Is it at all possible? How should I defined all network interfaces and what should be included in /etc/hosts for properly work of Oracle cluster?
Best Regards,
Jacek

Hi,
You can build a RAC witch CROSSOVER, but the Oracle no homolog.
As you have 4 cards, 2 to public (redundancy) and 2 to interconnect (redundancy) you need of a software to to make a TEAM, and create a card virtual that will have a IP address.
Eder

Dedicated switches needed for RAC interconnect or not?

Currently working on an Extended RAC cluster design implementation, I asked the network engineer for dedicated switches for the RAC interconnects.
Here is a little background:
There are 28 RAC clusters over 2X13 physical RAC nodes with separate Oracle_Home for each instance with atleast 2+ instances on each RAC node. So 13 RAC nodes will be in each site(Data-Center). This is basically an Extended RAC solution for SAP databases on RHEL 6 using ASM and Clusterware for Oracle 11gR2. The RAC nodes are Blades in a c7000 enclosure (in each site). The distance between the sites is 55+ kms.
Oracle recommends to have Infiniband(20GBps) as the network backbone, but here DWDM will be used with 2X10 Gbps (each at 10 GBps) links for the RAC interconnect between the sites. There will be separate 2x1GBps redundant link for the Production network and 2x2 GBps FC(Fiber-Channel) redundant links for the SAN/Storage(ASM traffic will go here) network. There will be switches for the Public-production network and the SAN network each.
Oracle recommends dedicated switches(which will give acceptable latency/bandwith) with switch redundancy to route the dedicated/non-routable VLANs for the RAC interconnect (private/heartbeat/global cache transfer) network. Since the DWDM interlinks is 2x10Gbps - do I still need the dedicated switches?
If yes, then how many?
Your inputs will be greatly appreciated.. and help me take a decision.
Many Thanks in advance..
Abhijit

Absolutely agree.. the chances of overload in a HA(RAC) solution and ultmate RAC node eviction are very high(with very high latency) and for exactly this reason I even suggested inexpensive switches to route the VLANs for the RAC interconnect through these switches. The ASM traffic will get routed through the 2x2GB FC links through SAN-Directors (1 in each site).
Suggested the network folks to use Up-links from the c7000 enclosure and route the RAC VLAN through these inexpensive switches for the interconnect traffic. We have another challenge here: HP has certified using VirtualConnect/Flex-Fabric architecture for Blades in c7000 to allocate VLANs for RAC interconnect. But this is only for one site, and does not span Production/DR sites separated over a distance.
Btw, do you have any standard switch model to select from.. and how many to go for a RAC configuration of 13 Extended RAC clusters with each cluster hosting 2+ RAC instances to host total of 28 SAP instances.
Many Thanks again!
Abhijit

INS-20802 Oracle Private Interconnect Configuration Assistant failed

Thought I would post what information I've gathered, after facing this error during install of RAC Grid Infrastructure 11.2.0.1 on Red Hat Enterprise Linux Server release 5.5 64-bit, as Oracle Support is once again unable to help. Maybe this will save someone else some time and the aggravation of dealing with lousy Oracle Support.
The error occurs after root.sh has successfully completed on all nodes. Oracle Net Configuration Assistant runs successfully, then Oracle Private Interconnect Configuration Assistant launches and subsequently fails with the following.
[INS-20802] Oracle Private Interconnect Configuration Assistant failed.
/u01/app/oraInventory/logs/installActions2010-12-13_01-26-10PM.log
INFO: Starting 'Oracle Private Interconnect Configuration Assistant'
INFO: Starting 'Oracle Private Interconnect Configuration Assistant'
INFO: PRIF-26: Error in update the profiles in the cluster
INFO:
WARNING:
INFO: Completed Plugin named: Oracle Private Interconnect Configuration Assistant
INFO: Oracle Private Interconnect Configuration Assistant failed.
INFO: Oracle Private Interconnect Configuration Assistant failed.
I was able to find another error that coincides with the PRIF-26 error: "CRS-2324:Error(s) occurred while trying to push GPnP profile. State may be inconsistent."
I was also able to duplicate the PRIF-26 error by trying to add a non-existent network interface via oifcfg:
./oifcfg setif -global jjj1/192.167.1.0:cluster_interconnect
PRIF-26: Error in update the profiles in the cluster
My best guess is the Oracle Private Interconnect Configuration Assistant makes a call to oifcfg. When oifcfg makes an update or adds a public/private interface, some XML files are also updated or maybe cross-referenced. These files are located here: <grid_home>/gpnp/<host>/profiles/peer
Any updates/changes/addtions to the private or public interfaces include changes for the Grid Plug-n-Play component, which uses the XML files. If the interface name is not contained in the XML files, my best guess is that triggers the "CRS-2324:Error(s) occurred while trying to push GPnP profile. State may be inconsistent.
I verified everything was configured correctly; the cluster verification utility reported everything was ok. I also ran the cluster verifcation utility against the GP-nP:
./cluvfy comp gpnp -verbose
I also reviewed the public and private interfaces via oifcfg and they are correct:
[oracle@ryoradb1 bin]$ ./oifcfg getif -global
eth0 10.10.2.0 global public
eth1 192.167.1.0 global cluster_interconnect
[oracle@ryoradb1 bin]$ ./oifcfg iflist -p
eth0 10.10.2.0 PRIVATE
eth1 192.167.1.0 UNKNOWN
My conclusion is the environment is configured correctly, in spite of the error generated by the Oracle Private Configuration Assistant.

I understand that you have installed 11.2.0.1 not 11.2.0.2 because multicating must be enabled if you have installed 11.2.0.2 and you may face these sort of problems because cluster nodes would not be able to communicate with each other.
Please check ocssd.log especially on the first node because this file will give more inforamtion as to why first node is not able to push GPnP file. As you have executed cluvfy to confirm the GPnP but to confirm whether GPnP profile is accurate to just narrow down the problem, I would suggest you to try to start cluster in exclusive mode so that you can sure that GPnP profile is good.
Shutdown CRS on all nodes, if there is any hung processes then kill them before executing the fllowing command to start cluster in exclusive mode.
$GRID_HOME/bin/crsctl start crs -excl
If you are able to start cluster in exclusive then it is sure that GPnP is correct and then next step would be to verify the private network.
See how you goes.
FYI, I was recently analyzing the same sort of problems where cluster was not able to access GPnP profile and finally I found issues on my private network. Customer had enabled IGMP snooping, which was avoiding multicast communication over the private network but it was 11.2.0.2, which is not the case here.
Harish Kumar
http://www.oraxperts.com

Crs Not Starting _ private Interconnect Down

Hello All,
I Have Installed 2 node 10g R2(10.2.0.1) RAC on Solaris 10 T2000 Machines. Yesterday my Second Node Crs gone down. I tried to start it but it didn't start. Then i checked that Private IP (interconnect) is not Pinging from both the node. But Node 1 was up and working so my Users Can Connect to It.
But Today morning I see that Crs on node 1aslo goes down .
Is this is problem of private interconnect.? My network guys are trying to up Private Interconnect.
If Private Interconnect is down, why node 1 goes down after few hours. i think private interconnect is for interconnect with node 2 but node 2 is down .
Previously My interconnect was connected with cross cables now i have asked them to connect them through switch.
Help me Out.
Regards,
Pankaj.

Previously My interconnect was connected with cross cables now i have asked them to connect them through switch
Even we are planning to do the same.Please share your experienceHope you have done this before - moving to switch
(Update for record id(s): 105681546)
QUESTION
========
1.Will the database and the Clusterware need to be shutdown etc?
2.Will our ip addresses need to be reconfigured?
3.Are there any steps that need to be carried out before unplugging the CROSS CABLE
and after the interconnect is connected to the switch...?
ANSWER
======
1. Yes, you have to stop CRS on each node.
2. No, not required.Provided you are planning to use same ip addresses.
3. Steps:
a. Stop CRS on each node. "crsctl stop crs"
b. Replace the crossover cable with switch.
c. Start the CRS on each node. "crsctl start crs"
Even we are planning to do the same.Please share your experienceFollowed by the above answers from Customer Support, It went smooth, we stopped all the services, and with both the nodes reboot.
Message was edited by:
Ravi Prakash

NICs for Private Interconnect redundancy

Similar Messages

Maybe you are looking for