Exadata extended RAC

Hi,
could we exented RAC between 2 exadatas ?
In this case how manage Voting Disk ?
Thank a lot
Edited by: 905160 on 30 déc. 2011 03:06

There are several points to consider about such a configuration:
1- Ib cables can go a 100m or so, so this is not really a stretch cluster, as in any decent data center, 100m apart means 2 rooms.
2- If you setup such a system, you need to configure ASM failure groups yourself, in a way to have the data from the first system mirrored on cells on the second system.
3- if you use normal redundancy, and you loose one rack, you'll end up with the second system having no redundancy at all, and you better rebalance right away to be on the safe side, this means keeping free space to be able to rebalance. If you can afford it, have 3 systems, with high redundancy, so that the los of one keeps the remaining part mirrored.
4- if you do it with 2 machines, better have a 3rd voting on an NFS server somewhere, like you would do with any stretch cluster.
That said, I do not like, nor advise such a setup at all. setting up a DG site for HA is far more easier, useful, robust, .....

Similar Messages

Any Benefits of Extended RAC Over Data Guard?

Hi,
My company is in the process of setting a second data center, a bit far from the current one (about 20 KM).
This Data Center will be used as a DR site, as well as to accommodate additional servers since the current Data Center is already stretched.
We're currently running about 5 RAC clusters, two nodes each on Oracle 11g and on AIX plaforms. It's not yet decided what type of technology will be employed with the databases - whether RAC with Data Guard (DG) or Extended RAC.The network link will be fairly good, a dark fiber link.
Does anyone have a suggestion as to which of the above technology would be preferable? With Extended RAC I think we are able to continue operating from the second site without needing to 'failover' as such, while with DG, we probably will need to put in place an elaborate failover procedure, even though we can use the Fast-Start-Automatic failover feature of 10g Rel2 and above.
Any thoughts/suggestions/clarifications?
Dula
Edited by: dula on Aug 28, 2009 2:47 PM

Hi Dula,
Even some time back we were also consdering the same to use remote DG for disaster recovery solution (active-passive) to create extended RAC on a remote site and then make use of those remote servers for the purpose of disaster reocovery and also make those servers connect to the PROD application servers thus making a (active-active) clustering solution.
Management wanted to look into the feasibility of using active-active nodes on remote sites which they thought will help them make use of remote servers to connect to the LIVE applicatio. With the active-passive mode where there was just standby servers in recovery mode on remote site they were of the opinion that why to waste power for remote servers and why not use them in LIVE site.
We thought of using the dense optic fiber for the DB replication to remote RAC nodes and also for our 30+ application servers to connect from site 1 to site 2( site 2 will be remote site with extented rac nodes hosted).
However there were many reservations that came which making decission. First was that we could not find any reliable source of group who had successfully implemented this active-active remote extended rac nodes in production. The cost of dense optic fiber was another consideration. How the interconnects will perform in remote sites was another grey area for us. Also since the application servers uses TAF to connect to DB servers it was not known that how the sessions from the same application servers(web based applications) to local and remote nodes will have performance impact. With so many grey areas we dropped the idea of using extended rac nodes and went ahead with DG solution.
Amar

More on Extended RAC

Erik Peterson (and others who might be interested),
I looked at your Sep '06 "Extended RAC" paper and thought it was very good. In it, you show data and case studies for distances up to 100km. In a post (Re: RAC on SUN geo clusters you say you would never do 1200km. At what point do you say "X kilometers is just too far"?
We have some folks talking about doing RAC over 256km with Dark Fibre. We can multiplex with 40 different frequencies, and tests have shown an average latency of 4ms. How do we decide whether to look at RAC or not? Obviously, other High Availability options are still appropriate (Active/Passive, Data Guard), but I wanted to field the question for the sake of gathering data. I realize some people would rightly question why in the world we are even thinking about this, so consider this post one mostly of curiosity and research. =)

Well it all a matter of what performance you are willing to deal with.
You would be adding your one way latency to each cache fusion message, and round trip latency to all disk I/Os as you will want disk to be synchronous.
How much this will affect an application will depend upon how much I/O and cache fusion bound it is.
That combined w/ the unknown factor of what performance degradation you are willing to live with makes it hard to put an absolute number. Things just get worse and worse.
Personally I am very comfortable implementing extended RAC in a metro area, anything beyond that I view first as an intiial estimate (OK are you willing to live w/ this degradation) followed by a serious POC (OK are you really, really willing to live w/ this degradation)
Hope this helps.
-Erik

VIP and Public Network on Extended Rac

Hi Everybody
I want to install Oracle Extended Rac on two Sites which is 25Km far and the spec is :
Oracle 11.1.0.6
HP PA-RISC server
EVA Storage
my problem is
each of my sites has different subnet for public network so for VIP failover there is a problem.
how can solve this problem.
can we have two site with on subnet? how?
I have this problem eith Interconnect too.
thanks anyway
Hashem

hi
mind oifcfg tool
http://www.databasejournal.com/features/oracle/article.php/3651826/Oracle-RAC-Administration---Part-12-RAC-Essentials.htm
and
http://download.oracle.com/docs/cd/B28359_01/rac.111/b28255/oifcfg.htm

Extended RAC failure question

Hi gurus.
I have a question about this kind of environment.
In a Extended RAC scenario, with host-based mirroring (active/active storage with ASM) and a third site to mantain the voting disk, What will happen in a communication failure between the two sites (DWDM failure)?
This failure could be a problem? The Oracle RAC will survive?
Thank you!

Hi,
In a Extended RAC scenario, with host-based mirroring (active/active storage with ASM) and a third site to mantain the voting disk, What will happen in a communication failure between the two sites (DWDM failure)?
This failure could be a problem? The Oracle RAC will survive?This questions you must not get here ... you must get these answers by testing their environment.
Extended clusters need additional destructive testing, covering:
** Site failure*
** Communication failure*
But..... If DWDM fails the split-brain scenario should happen. You should search about "Oracle RAC Cluster Fencing ( For your specific release)" to understand the symptoms caused by split-brain scenario.
Regards,
Levi Pereira

10g Extended RAC with Symantec/Veritas

Hi,
Do you know any solution for an active-active 10g extended RAC using Symantec/Veritas Storage Foundation ?
I've seen some active-pasive solutions using Veritas/Symantec GCO and VVR but we would want that the extended node can be online and do some workload.
Thanks in advance
Regards
Jose

You might take campus cluster into your consideration. (2 SFRAC clusters + VxVM Mirror)
Or you can also consitder : 2 SFRAC clusters + GCO + Logical Oracle dataguard.
Thanks.

Using dbca to extend RAC cluster error

Hi all,
I'm trying to extend my 11gR2 RAC cluster (POC) using the Oracle documentation (http://vishalgupta.com/oracle/docs/Database11.2/rac.112/e10718/adddelunix.htm). I've already cloned and extended Clusterware and ASM (Grid Infrastructure) to the new node, as well as cloned the RAC database software to the new node. When I run the below statement to have dbca extend add a new instance on the node for the RAC I get the error shown:
CMD:
$ORACLE_HOME/bin/dbca -silent -addInstance -nodeList newnode13 -gdbName racdb -instanceName racdb4 -sysDBAUserName sys
-sysDBAPassword manager123
ERROR:
cat racdb0.log
"Adding instance" operation on the admin managed database racdb requires instance configured on local node. There is no instance configured on the local node "newnode13".
I set ORACLE_HOME before running dbca, and I've also tried setting ORACLE_SID to both racdb4 and racdb, no change. My environment is below, any help is appreciated.
OS: SLES 11.1
Database: 11.2.0.1
Existing Nodes: node01,node02, node03
New Node: newnode13
DB Name: racdb
Instances: racdb1, racdb2, racdb3
New Instance: racdb4
Thanks.

Silly me, I was running the command from the new node instead of an existing node. I guess it was a rough weekend after all. Thanks all!

Extended RAC connectivity.

We are thinking extend our RAC in one kilometer.
For interconnect (in the same room) we are using a 1Gb channel and 2Gb (fiber) to the SAN.
In order to extend our RAC in one kilometer using "Remote Array-Based Mirroring" based on two SAN, do we need to think in dark fiber and DWDM for interconnect and storage access or in opposite for this distance could we mantain our actual infrastructure?
Thanks in advance,
Diego.

What are you trying to accomplish by doing this? IF this works, there are lots of costs involved. Mirroring SANs to have 1 RAC'ed database w/ nodes 1KM apart using the different SANs to write to the database doesn't sound like it would work right. Even if it does work, what would happen to transactions that need to failover to the other nodes if the fiber channel goes down?
Something I always tell people when they are trying to architect complicated structures, just because you can doesn't mean you should....

Multiple databases/instances on 4-node RAC Cluster including Physical Stand

OS: Windows 2003 Server R2 X64
DB: 10.2.0.4
Virtualization: NONE
Node Configuration: x64 architecture - 4-Socket Quad-Core (16 CPUs)
Node Memory: 128GB RAM
We are planning the following on the above-mentioned 4-node RAC cluster:
Node 1: DB1 with instanceDB11 (Active-Active: Load-balancing & Failover)
Node 2: DB1 with instanceDB12 (Active-Active: Load-balancing & Failover)
Node 3: DB1 with instanceDB13 (Active-Passive: Failover only) + DB2 with instanceDB21 (Active-Active: Load-balancing & Failover) + DB3 with instanceDB31 (Active-Active: Load-balancing & Failover) + DB4 with instance41 (Active-Active: Load-balancing & Failover)
Node 4: DB1 with instanceDB14 (Active-Passive: Failover only) + DB2 with instanceDB22 (Active-Active: Load-balancing & Failover) + DB3 with instanceDB32 (Active-Active: Load-balancing & Failover) + DB4 with instance42 (Active-Active: Load-balancing & Failover)
Note: DB1 will be the physical primary PROD OLTP database and will be open in READ-WRITE mode 24x7x365.
Note: DB2 will be a Physical Standby of DB1 and will be open in Read-Only mode for reporting purposes during the day-time, except for 3 hours at night when it will apply the logs.
Note: DB3 will be a Physical Standby of a remote database DB4 (not part of this cluster) and will be mounted in Managed Recovery mode for automatic failover/switchover purposes.
Note: DB4 will be the physical primary Data Warehouse DB.
Note: Going to 11g is NOT an option.
Note: Data Guard broker will be used across the board.
Please answer/advise of the following:
1. Is the above configuration supported and why so? If not, what are the alternatives?
2. Is the above configuration recommended and why so? If not, what are the recommended alternatives?

Hi,
As far as i understand, there's nothing wrong in configuration except you need to consider below points while implementing final design.
1. No of CPU on each servers
2. Memory on each servers
3. If you've RAC physical standby then apply(MRP0) will run on only one instance.
4. Since you are configuring physical standby for on 3rd and 4th nodes of DB1 4 node cluster where DB13 and DB14 instances are used only for failver, if you've a disaster at data center or power failure in entire data center, you are losing both primary and secondary with an assumption that your primary and physical standby reside in same data center so it may not be highly available architecture. If you are going to use extended RAC for this configuration then it makes sense where Node 1 and Node 2 will reside in Datacenter A and Node 3 ,4 will reside in Datacenter B.
Thanks,
Keyur

Question on RAC

Hi friends,
I have got this question from customer regarding Oracle 11G R2 RAC.
The customer have stated a preference for configuring the Oracle RAC nodes so that, in a two node system, the first node bears all the update load, whereas the second RAC node would have connections that are read-only. The two nodes would still operate as failovers for one another. The reason for this , they stated was to reduce network traffic between the nodes to coordinate transactions and locking. They have asked for this configuration as 'one update node and one read-only node' / a primary update node and a secondary read-only node .
We have thought about implementing this configuration for our two applications as follows
Application features
Application 1 - which performs regular updates,heavy transactions,long running reports et - schema 1
Application 2 - performs smalled updates and small reads. - schema 2
Application 1 can point to node 1 of this RAC and application 2 to node 2 ( using node affinity ).When any of the node fails , the client connections will fail over to surviving node.
I would like to get some recommendations on this approach from experts.
1) Do you think the above approach using node affinity is a good approach , ie pinning one schema in one node with one database for a two instance(node) RAC ?
2) Any documents / reference to configure this approach in RAC ?
3) Is there any other way to achieve the requirement put by the customer ?
4) If the above said nodes are in two data centres, Will it be a recommended approach ? I think these two nodes should be in same data centre.
5) If we configure Load balancing ( in normal RAC or in the setup mentioned above) then that would take considerable network b/w ?
OS is RHEL 5.4 on 64 bit.
Please share your view on this.
Thanks ,
SSN

1) Do you think the above approach using node affinity is a good approach , ie pinning one schema in one node with one database for a two instance(node) RAC ?In RAC , database remains same. So all schemas can be accessed from all nodes.
You can configure RAC services and then you can use one service for the updates and the other one can be used for read only application.
http://download.oracle.com/docs/cd/E11882_01/rac.112/e16795/hafeats.htm#BABJJJAG
4) If the above said nodes are in two data centres, Will it be a recommended approach ? I think these two nodes should be in same data centre.Why do you want to put RAC nodes in different data centers ?
You can use Oracle extended RAC in that case.
http://blogs.oracle.com/AlejandroVargas/2008/03/extended_rac_on_pure_oracle_st.html
http://www.databasejournal.com/features/oracle/article.php/3675096/Oracle-Extended-RAC-Stretching-your-RAC-to-the-limit.htm
Regards
Rajesh

Oracle 11gR2 RAC setup with ASM option running on two nodes . Seeking help

Hi ,
Recently we have migrated to 11gR2 extended RAC for one of our application using ASM . Its a two node RAC setup . We are using ASM Normal redundancy feature maintaining two failure group .
I need few clarification regarding our new DR set up .
We are planning to have EMC SRDF replication from any of node storage ( Either 1 or 2) to the DR disks .
Its stright to use when we use the normal file systems . In case of ASM , I am not sure how this replication will work ? and on which level the replication should happen ?
Do i need an ASM instance up and running on DR server , Which will recognize the disks as it is once DR disks are separated from prod replication and assigned to the server ?
What should be prerequisite on DR server before we can start the srdf replication (EMC) and make it work .
Thanks for All your Valuble suggestion .

Before you decide to use EMC replication give considerable thought to the fact that you are introducing a single point of failure.
Hardware snapshot replication is very fast. That is all it is. If you database crashes because of corruption on disk ... your replica will be equally corrupt.
What is fast can not do what DataGuard does ... verify quality. If the point is DR. Go with DataGuard unless EMC is willing to, in writing (not verbally) give you a guarantee (in hard cash) to cover all costs if they replicate bad blocks and it causes an outage.
And just so no one thinks I'm picking o EMC ... the same weakness exists with all of the other storage vendor's products.

Manually Setting up a Standby Database for RAC Environment

Hi,
I am a Junior DBA so apologies if this sounds like a silly question.
We have production 11g Extended RAC environment setup. Management have asked for a separate single instance database be setup as a standby DB on a separate 'cold' server in case we lose complete connection to both sides of the RAC. I read up on Oracle Data Guard and presented this as a solution, but they seem adamant about manually going in and copying out the latest backup and archive logs over to the standby database.
Can this be done? I mean, can ASM managed backup files and archive logs from one database be simply copied out of the backup directory and imported into a completely separate database as easy as that?
We are using ASM to manage to data files on the RAC. My understanding is that we cant manually access the files on an OS level when using ASM, but maybe I am wrong. Any help or opinions on this would be greatly appreciated.
Rgs,
Rob

Hi,
Can this be done? I mean, can ASM managed backup files and archive logs from one database be simply copied out of the backup directory and imported into a completely separate >database as easy as that?Yes, but depends on DB version.
what is the DB version.
from 11g you have cp command so there is a possibility to copy generated archive logs to standby location, so that you can apply..
In 10g there is no such option. you need to create a standby database. with automatic shipping
Thanks

Exadata + Physical Standby (non-exadata) + Backups

Some folks are asking me for my opinion on a backup strategy - I don't think its possible/feasible, but I will put it out here for comments..
Proposal:
Create physical dg db (non exadata) with netapp storage of an Exadata (full rac v2).
Create rman backups of the physical dg using netapp snapshot technologies.
I cant see how this would work. Sure we can take the snaps on the pdg and restore the pdg, but ..
a) what if we lose a datafile on the exadata?
Its asm - what rman command would you run to restore the datafile? a simple restore datafile? I would think it would be more complicated.
b) what if we lost all of the exadata - corruption, physical,logical whatever..
rman restore database wouldn't work would it? Would it know to restore from the backups done on the pdg?
EIther way - it seems that you are inevitably restoring upwards of ~100tb from the pdg back to the Exadata via some pipe (nic/Ib).
I see one comment that says you can use pdg backups if they went to tape, but not if they went to disk. I assume the netapp snaps would be considered disk backups.
Thoughts?

Objectives -- quick restore times (< 8hrs) - not to rely on tapes - full backups - reduce impact on source db
Size being upwards of 50-100Tb
The vendor has outlined the product which "seems" like a great thing - but will it work with Exadata
See this thread on the RAC-RMAN with snapshot on NetApp

Dedicated switches needed for RAC interconnect or not?

Currently working on an Extended RAC cluster design implementation, I asked the network engineer for dedicated switches for the RAC interconnects.
Here is a little background:
There are 28 RAC clusters over 2X13 physical RAC nodes with separate Oracle_Home for each instance with atleast 2+ instances on each RAC node. So 13 RAC nodes will be in each site(Data-Center). This is basically an Extended RAC solution for SAP databases on RHEL 6 using ASM and Clusterware for Oracle 11gR2. The RAC nodes are Blades in a c7000 enclosure (in each site). The distance between the sites is 55+ kms.
Oracle recommends to have Infiniband(20GBps) as the network backbone, but here DWDM will be used with 2X10 Gbps (each at 10 GBps) links for the RAC interconnect between the sites. There will be separate 2x1GBps redundant link for the Production network and 2x2 GBps FC(Fiber-Channel) redundant links for the SAN/Storage(ASM traffic will go here) network. There will be switches for the Public-production network and the SAN network each.
Oracle recommends dedicated switches(which will give acceptable latency/bandwith) with switch redundancy to route the dedicated/non-routable VLANs for the RAC interconnect (private/heartbeat/global cache transfer) network. Since the DWDM interlinks is 2x10Gbps - do I still need the dedicated switches?
If yes, then how many?
Your inputs will be greatly appreciated.. and help me take a decision.
Many Thanks in advance..
Abhijit

Absolutely agree.. the chances of overload in a HA(RAC) solution and ultmate RAC node eviction are very high(with very high latency) and for exactly this reason I even suggested inexpensive switches to route the VLANs for the RAC interconnect through these switches. The ASM traffic will get routed through the 2x2GB FC links through SAN-Directors (1 in each site).
Suggested the network folks to use Up-links from the c7000 enclosure and route the RAC VLAN through these inexpensive switches for the interconnect traffic. We have another challenge here: HP has certified using VirtualConnect/Flex-Fabric architecture for Blades in c7000 to allocate VLANs for RAC interconnect. But this is only for one site, and does not span Production/DR sites separated over a distance.
Btw, do you have any standard switch model to select from.. and how many to go for a RAC configuration of 13 Extended RAC clusters with each cluster hosting 2+ RAC instances to host total of 28 SAP instances.
Many Thanks again!
Abhijit

RAC database Migration

Hi Experts,
I need some advice related to the Migration of RAC database, we have to migrate several RAC & Single-Instance databases from existing HP Itanium servers to Superdome2.
Initially I thought of going with "addnode" for new Servers as additional Nodes to the existing RAC database then remove old Nodes for RAC databases migration, but there are some limitations in our evironment,
1. There are no free IP addresses available to assign to new nodes from the existing IP segment.
The IP Range (1 - 255) is divided into into several small segments of 16 IPs in each segment, and the segment in which the existing 2Node RAC DB is completely occupied and there is no free IP Available.
If the new Node have IPs addresses from different segment in the same Subnet(IP Range) can it be added to the existing cluster?
and if we cannot find any IP in that Subnet then can the nodes with the IPs from other Subnet be added to the Existing Cluster?
2. To add a node to the existing RAC cluster, the new node should be exactly same, OS version, OS Patches...
Our existing rx Itanium servers have "*HP-UX 11.31 Mission Critical Operating Environment*" for One 2 Node RAC Cluster and "*HP-UX 11.31 Enterprise Operating Environment*" for for another 2 Node RAC Cluster, but the HP Superdome 2 will have the Latest "*HP-UX 11.31 Data Center Operating Environment*"
Can we proceed if there are some diferrences in OS patches...?
Please see the Existing Environment Details below:*
Prod 1 DB *2 Node RAC Cluster* :
Hardware: rx 8640 Itanium Server
OS: HP-UX 11.31 v3 Mission Critical Operating Environment
DB Version: Oracle 10.2.0.2
Prod 2 DB *2 Node RAC Cluster*:
Hardware: rx 8640 Itanium Server
OS: HP-UX 11.31 v3 Enterprise Operating Environment
DB Version: Oracle 11.2.0.1 (2 Node RAC)
Prod 3 DB *2 Node RAC Cluster*:
Hardware: rx 6600 Itanium Server
OS: HP-UX 11.31 v3 Enterprise Operating Environment
DB Version: Oracle 11.1.0.6.0 (2 Node RAC)
TARGET DB Environment for All the Databases will be;*
Hardware: HP SUperdome 2
OS: HP-UX 11.31Data Center Operating Environment
DB Version: Oracle 11.2.0.3 (2 Node RAC)
Also, all the 3 RAC databases are in Extended/Stretch RAC Cluster configuration and the new Superdome Env will also be in Extended RAC Cluster Configuration.
Kindly advice what strategy should I follow for the Migration, waiting for your kind reply.
Any help will be highly appreciated.
Thanks in Advance.

I am struck by your post with a couple of thoughts.
1. Even though high availability appears to be something you are willing to invest substantial money in obtaining ... no one think patching important.
2. Even though you've spent a substantial sum of money on hardware and software ... no one thinks networking important enough to address the lack of appropriate network infrastructure.
Personally: I'd advise refusing to do the work until management makes the very small investment necessary to doing it correctly.

Exadata extended RAC

Similar Messages

Maybe you are looking for