More on Extended RAC

Erik Peterson (and others who might be interested),
I looked at your Sep '06 "Extended RAC" paper and thought it was very good. In it, you show data and case studies for distances up to 100km. In a post (Re: RAC on SUN geo clusters you say you would never do 1200km. At what point do you say "X kilometers is just too far"?
We have some folks talking about doing RAC over 256km with Dark Fibre. We can multiplex with 40 different frequencies, and tests have shown an average latency of 4ms. How do we decide whether to look at RAC or not? Obviously, other High Availability options are still appropriate (Active/Passive, Data Guard), but I wanted to field the question for the sake of gathering data. I realize some people would rightly question why in the world we are even thinking about this, so consider this post one mostly of curiosity and research. =)

Well it all a matter of what performance you are willing to deal with.
You would be adding your one way latency to each cache fusion message, and round trip latency to all disk I/Os as you will want disk to be synchronous.
How much this will affect an application will depend upon how much I/O and cache fusion bound it is.
That combined w/ the unknown factor of what performance degradation you are willing to live with makes it hard to put an absolute number. Things just get worse and worse.
Personally I am very comfortable implementing extended RAC in a metro area, anything beyond that I view first as an intiial estimate (OK are you willing to live w/ this degradation) followed by a serious POC (OK are you really, really willing to live w/ this degradation)
Hope this helps.
-Erik

Similar Messages

Any Benefits of Extended RAC Over Data Guard?

Hi,
My company is in the process of setting a second data center, a bit far from the current one (about 20 KM).
This Data Center will be used as a DR site, as well as to accommodate additional servers since the current Data Center is already stretched.
We're currently running about 5 RAC clusters, two nodes each on Oracle 11g and on AIX plaforms. It's not yet decided what type of technology will be employed with the databases - whether RAC with Data Guard (DG) or Extended RAC.The network link will be fairly good, a dark fiber link.
Does anyone have a suggestion as to which of the above technology would be preferable? With Extended RAC I think we are able to continue operating from the second site without needing to 'failover' as such, while with DG, we probably will need to put in place an elaborate failover procedure, even though we can use the Fast-Start-Automatic failover feature of 10g Rel2 and above.
Any thoughts/suggestions/clarifications?
Dula
Edited by: dula on Aug 28, 2009 2:47 PM

Hi Dula,
Even some time back we were also consdering the same to use remote DG for disaster recovery solution (active-passive) to create extended RAC on a remote site and then make use of those remote servers for the purpose of disaster reocovery and also make those servers connect to the PROD application servers thus making a (active-active) clustering solution.
Management wanted to look into the feasibility of using active-active nodes on remote sites which they thought will help them make use of remote servers to connect to the LIVE applicatio. With the active-passive mode where there was just standby servers in recovery mode on remote site they were of the opinion that why to waste power for remote servers and why not use them in LIVE site.
We thought of using the dense optic fiber for the DB replication to remote RAC nodes and also for our 30+ application servers to connect from site 1 to site 2( site 2 will be remote site with extented rac nodes hosted).
However there were many reservations that came which making decission. First was that we could not find any reliable source of group who had successfully implemented this active-active remote extended rac nodes in production. The cost of dense optic fiber was another consideration. How the interconnects will perform in remote sites was another grey area for us. Also since the application servers uses TAF to connect to DB servers it was not known that how the sessions from the same application servers(web based applications) to local and remote nodes will have performance impact. With so many grey areas we dropped the idea of using extended rac nodes and went ahead with DG solution.
Amar

VIP and Public Network on Extended Rac

Hi Everybody
I want to install Oracle Extended Rac on two Sites which is 25Km far and the spec is :
Oracle 11.1.0.6
HP PA-RISC server
EVA Storage
my problem is
each of my sites has different subnet for public network so for VIP failover there is a problem.
how can solve this problem.
can we have two site with on subnet? how?
I have this problem eith Interconnect too.
thanks anyway
Hashem

hi
mind oifcfg tool
http://www.databasejournal.com/features/oracle/article.php/3651826/Oracle-RAC-Administration---Part-12-RAC-Essentials.htm
and
http://download.oracle.com/docs/cd/B28359_01/rac.111/b28255/oifcfg.htm

Extended RAC failure question

Hi gurus.
I have a question about this kind of environment.
In a Extended RAC scenario, with host-based mirroring (active/active storage with ASM) and a third site to mantain the voting disk, What will happen in a communication failure between the two sites (DWDM failure)?
This failure could be a problem? The Oracle RAC will survive?
Thank you!

Hi,
In a Extended RAC scenario, with host-based mirroring (active/active storage with ASM) and a third site to mantain the voting disk, What will happen in a communication failure between the two sites (DWDM failure)?
This failure could be a problem? The Oracle RAC will survive?This questions you must not get here ... you must get these answers by testing their environment.
Extended clusters need additional destructive testing, covering:
** Site failure*
** Communication failure*
But..... If DWDM fails the split-brain scenario should happen. You should search about "Oracle RAC Cluster Fencing ( For your specific release)" to understand the symptoms caused by split-brain scenario.
Regards,
Levi Pereira

10g Extended RAC with Symantec/Veritas

Hi,
Do you know any solution for an active-active 10g extended RAC using Symantec/Veritas Storage Foundation ?
I've seen some active-pasive solutions using Veritas/Symantec GCO and VVR but we would want that the extended node can be online and do some workload.
Thanks in advance
Regards
Jose

You might take campus cluster into your consideration. (2 SFRAC clusters + VxVM Mirror)
Or you can also consitder : 2 SFRAC clusters + GCO + Logical Oracle dataguard.
Thanks.

Using dbca to extend RAC cluster error

Hi all,
I'm trying to extend my 11gR2 RAC cluster (POC) using the Oracle documentation (http://vishalgupta.com/oracle/docs/Database11.2/rac.112/e10718/adddelunix.htm). I've already cloned and extended Clusterware and ASM (Grid Infrastructure) to the new node, as well as cloned the RAC database software to the new node. When I run the below statement to have dbca extend add a new instance on the node for the RAC I get the error shown:
CMD:
$ORACLE_HOME/bin/dbca -silent -addInstance -nodeList newnode13 -gdbName racdb -instanceName racdb4 -sysDBAUserName sys
-sysDBAPassword manager123
ERROR:
cat racdb0.log
"Adding instance" operation on the admin managed database racdb requires instance configured on local node. There is no instance configured on the local node "newnode13".
I set ORACLE_HOME before running dbca, and I've also tried setting ORACLE_SID to both racdb4 and racdb, no change. My environment is below, any help is appreciated.
OS: SLES 11.1
Database: 11.2.0.1
Existing Nodes: node01,node02, node03
New Node: newnode13
DB Name: racdb
Instances: racdb1, racdb2, racdb3
New Instance: racdb4
Thanks.

Silly me, I was running the command from the new node instead of an existing node. I guess it was a rough weekend after all. Thanks all!

Exadata extended RAC

Hi,
could we exented RAC between 2 exadatas ?
In this case how manage Voting Disk ?
Thank a lot
Edited by: 905160 on 30 déc. 2011 03:06

There are several points to consider about such a configuration:
1- Ib cables can go a 100m or so, so this is not really a stretch cluster, as in any decent data center, 100m apart means 2 rooms.
2- If you setup such a system, you need to configure ASM failure groups yourself, in a way to have the data from the first system mirrored on cells on the second system.
3- if you use normal redundancy, and you loose one rack, you'll end up with the second system having no redundancy at all, and you better rebalance right away to be on the safe side, this means keeping free space to be able to rebalance. If you can afford it, have 3 systems, with high redundancy, so that the los of one keeps the remaining part mirrored.
4- if you do it with 2 machines, better have a 3rd voting on an NFS server somewhere, like you would do with any stretch cluster.
That said, I do not like, nor advise such a setup at all. setting up a DG site for HA is far more easier, useful, robust, .....

Extended RAC connectivity.

We are thinking extend our RAC in one kilometer.
For interconnect (in the same room) we are using a 1Gb channel and 2Gb (fiber) to the SAN.
In order to extend our RAC in one kilometer using "Remote Array-Based Mirroring" based on two SAN, do we need to think in dark fiber and DWDM for interconnect and storage access or in opposite for this distance could we mantain our actual infrastructure?
Thanks in advance,
Diego.

What are you trying to accomplish by doing this? IF this works, there are lots of costs involved. Mirroring SANs to have 1 RAC'ed database w/ nodes 1KM apart using the different SANs to write to the database doesn't sound like it would work right. Even if it does work, what would happen to transactions that need to failover to the other nodes if the fiber channel goes down?
Something I always tell people when they are trying to architect complicated structures, just because you can doesn't mean you should....

RAC on Windows Server 2008 R2 - which edition is necessary

Hello,
which Windows Server 2008 R2 edition is required to install RAC - standard or enterprise (NOT database edition - windows server 2008 edition)?
If we want to implement RAC 11Gr2 Enterprise Edition does the operating system also has to be enterprise edition or is it possible to install RAC 11GR2 Enterprise Edition on Windows Server 2008 Standard Edition?
What Server Version is more suited for RAC Windows Server 2003 or Windows Server 2008, and why?
Thanks in advance for any information.
kind regards
Monika Anna

Hi Monika,
which Windows Server 2008 R2 edition is required to install RAC - standard or enterprise (NOT database edition - windows server 2008 edition)?
If we want to implement RAC 11Gr2 Enterprise Edition does the operating system also has to be enterprise edition or is it possible to install RAC 11GR2 Enterprise Edition on Windows Server 2008 Standard Edition?All Oracle Database 11g Release 2 is supported on the following Windows Server 2008 R2 editions:
Windows Web Server 2008 (x64)
Windows Server 2008 Foundation (x64)
Windows Server 2008 Standard (x64)
Windows Server 2008 Enterprise (x64)
Windows Server 2008 Datacenter (x64)
What Server Version is more suited for RAC Windows Server 2003 or Windows Server 2008, and why?I recommend Windows 2008, because the system is newer and has a lot of features that do not exist in Windows 2003. Oracle 11.2 is certified in Windows 2008.
Check this note...
*Statement of Direction: Oracle Database 11g Release 2 – Microsoft Windows 7 and Windows Server 2008 R2 [ID 867040.1]*
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=ANNOUNCEMENT&id=867040.1
Regards,
Levi Pereira

Multiple databases/instances on 4-node RAC Cluster including Physical Stand

OS: Windows 2003 Server R2 X64
DB: 10.2.0.4
Virtualization: NONE
Node Configuration: x64 architecture - 4-Socket Quad-Core (16 CPUs)
Node Memory: 128GB RAM
We are planning the following on the above-mentioned 4-node RAC cluster:
Node 1: DB1 with instanceDB11 (Active-Active: Load-balancing & Failover)
Node 2: DB1 with instanceDB12 (Active-Active: Load-balancing & Failover)
Node 3: DB1 with instanceDB13 (Active-Passive: Failover only) + DB2 with instanceDB21 (Active-Active: Load-balancing & Failover) + DB3 with instanceDB31 (Active-Active: Load-balancing & Failover) + DB4 with instance41 (Active-Active: Load-balancing & Failover)
Node 4: DB1 with instanceDB14 (Active-Passive: Failover only) + DB2 with instanceDB22 (Active-Active: Load-balancing & Failover) + DB3 with instanceDB32 (Active-Active: Load-balancing & Failover) + DB4 with instance42 (Active-Active: Load-balancing & Failover)
Note: DB1 will be the physical primary PROD OLTP database and will be open in READ-WRITE mode 24x7x365.
Note: DB2 will be a Physical Standby of DB1 and will be open in Read-Only mode for reporting purposes during the day-time, except for 3 hours at night when it will apply the logs.
Note: DB3 will be a Physical Standby of a remote database DB4 (not part of this cluster) and will be mounted in Managed Recovery mode for automatic failover/switchover purposes.
Note: DB4 will be the physical primary Data Warehouse DB.
Note: Going to 11g is NOT an option.
Note: Data Guard broker will be used across the board.
Please answer/advise of the following:
1. Is the above configuration supported and why so? If not, what are the alternatives?
2. Is the above configuration recommended and why so? If not, what are the recommended alternatives?

Hi,
As far as i understand, there's nothing wrong in configuration except you need to consider below points while implementing final design.
1. No of CPU on each servers
2. Memory on each servers
3. If you've RAC physical standby then apply(MRP0) will run on only one instance.
4. Since you are configuring physical standby for on 3rd and 4th nodes of DB1 4 node cluster where DB13 and DB14 instances are used only for failver, if you've a disaster at data center or power failure in entire data center, you are losing both primary and secondary with an assumption that your primary and physical standby reside in same data center so it may not be highly available architecture. If you are going to use extended RAC for this configuration then it makes sense where Node 1 and Node 2 will reside in Datacenter A and Node 3 ,4 will reside in Datacenter B.
Thanks,
Keyur

Question on RAC

Hi friends,
I have got this question from customer regarding Oracle 11G R2 RAC.
The customer have stated a preference for configuring the Oracle RAC nodes so that, in a two node system, the first node bears all the update load, whereas the second RAC node would have connections that are read-only. The two nodes would still operate as failovers for one another. The reason for this , they stated was to reduce network traffic between the nodes to coordinate transactions and locking. They have asked for this configuration as 'one update node and one read-only node' / a primary update node and a secondary read-only node .
We have thought about implementing this configuration for our two applications as follows
Application features
Application 1 - which performs regular updates,heavy transactions,long running reports et - schema 1
Application 2 - performs smalled updates and small reads. - schema 2
Application 1 can point to node 1 of this RAC and application 2 to node 2 ( using node affinity ).When any of the node fails , the client connections will fail over to surviving node.
I would like to get some recommendations on this approach from experts.
1) Do you think the above approach using node affinity is a good approach , ie pinning one schema in one node with one database for a two instance(node) RAC ?
2) Any documents / reference to configure this approach in RAC ?
3) Is there any other way to achieve the requirement put by the customer ?
4) If the above said nodes are in two data centres, Will it be a recommended approach ? I think these two nodes should be in same data centre.
5) If we configure Load balancing ( in normal RAC or in the setup mentioned above) then that would take considerable network b/w ?
OS is RHEL 5.4 on 64 bit.
Please share your view on this.
Thanks ,
SSN

1) Do you think the above approach using node affinity is a good approach , ie pinning one schema in one node with one database for a two instance(node) RAC ?In RAC , database remains same. So all schemas can be accessed from all nodes.
You can configure RAC services and then you can use one service for the updates and the other one can be used for read only application.
http://download.oracle.com/docs/cd/E11882_01/rac.112/e16795/hafeats.htm#BABJJJAG
4) If the above said nodes are in two data centres, Will it be a recommended approach ? I think these two nodes should be in same data centre.Why do you want to put RAC nodes in different data centers ?
You can use Oracle extended RAC in that case.
http://blogs.oracle.com/AlejandroVargas/2008/03/extended_rac_on_pure_oracle_st.html
http://www.databasejournal.com/features/oracle/article.php/3675096/Oracle-Extended-RAC-Stretching-your-RAC-to-the-limit.htm
Regards
Rajesh

Oracle 11gR2 RAC setup with ASM option running on two nodes . Seeking help

Hi ,
Recently we have migrated to 11gR2 extended RAC for one of our application using ASM . Its a two node RAC setup . We are using ASM Normal redundancy feature maintaining two failure group .
I need few clarification regarding our new DR set up .
We are planning to have EMC SRDF replication from any of node storage ( Either 1 or 2) to the DR disks .
Its stright to use when we use the normal file systems . In case of ASM , I am not sure how this replication will work ? and on which level the replication should happen ?
Do i need an ASM instance up and running on DR server , Which will recognize the disks as it is once DR disks are separated from prod replication and assigned to the server ?
What should be prerequisite on DR server before we can start the srdf replication (EMC) and make it work .
Thanks for All your Valuble suggestion .

Before you decide to use EMC replication give considerable thought to the fact that you are introducing a single point of failure.
Hardware snapshot replication is very fast. That is all it is. If you database crashes because of corruption on disk ... your replica will be equally corrupt.
What is fast can not do what DataGuard does ... verify quality. If the point is DR. Go with DataGuard unless EMC is willing to, in writing (not verbally) give you a guarantee (in hard cash) to cover all costs if they replicate bad blocks and it causes an outage.
And just so no one thinks I'm picking o EMC ... the same weakness exists with all of the other storage vendor's products.

Oracle 11gR2 RAC installation with DNS

Dear Guru's,
I want to configure RAC on two node cluster with DNS(For SCAN) in my testing server.
O.S: RHEL5.4(64 bit)
Oracle Version: 11gR2(11.2.0.2)
Note: Am creating nodes in VMware ESXi server.
Just I want to know whether DNS should be configure in any one of those NODES or in separate machine.
Bala :)

Hi Bala,
I want to configure my Database with & without DNS. (Will try both installation in different servers)I think, you have already received a link on very good article about Oracle RAC installation without DNS and DHCP (I mean Jeffrey Hunter's article).
There is one more about Oracle RAC installation with DNS and DHCP (http://gjilevski.com/2011/10/05/build-two-node-oracle-rac-11gr2-11-2-0-3-with-gns-dns-dhcp-and-haip/).
And want to know how it forwards the client connection if a node fails.. Can anyone update the URL for this..There are a lot of information about it:
- http://www.oracle.com/technetwork/database/features/oci/taf-10-133239.pdf
- http://docs.oracle.com/cd/E14072_01/java.112/e10589/apxracfan.htm
- http://www.oracle.com/technetwork/database/app-failover-oracle-database-11g-173323.pdf
Very good videos:
- http://www.dsvolk.ru/oracle/racdd4d/demos/video/taf/session/session_viewlet_swf.html
- http://www.dsvolk.ru/oracle/racdd4d/demos/video/taf/select/select_viewlet_swf.html
- http://www.dsvolk.ru/oracle/racdd4d/demos/video/taf/callback/callback_viewlet_swf.html
- http://www.dsvolk.ru/oracle/racdd4d/demos/video/fan/fan_viewlet_swf.html
Hope it helps,
Best regards,
Gena

User equivalence in RAC

This is a question regarding resetting the user equivalence on clustered hosts. Our company has a 3-node 10.2.0.3 RAC cluster running CRS w/ an ASM system for the DB (Linux64). I just finished rebuilding our Grid Control deployment on a new host (bringing it up to 10.2.0.5) and have been steadily redeploying the agents. The last boxes on the list were those associated with the production RAC cluster. During installation, you have the option to install to all 3 nodes simultaneously from a single node, however when I attempted this, I was informed that password-less ssh communication between the 3 nodes was not available. Not a big problem since I just went through the installs on each box locally and each host and its targets are now reporting to the Grid Control quite well. It even properly detects that the database is clustered.
However, a Grid critical alert continues to inform me that "User equivalence unavailable on all the nodes", and I decided to investigate the possibilities of rectifying this. Initially I tried copying appropriate files out of the ~/.ssh directory to other boxes, but that's when I discovered that the keys had been pass-phrased and neither myself nor the other DBA here know what that pass-phrase is (accurate, centrally managed password management was something I put in place when I arrived to prevent this very thing going forward - thank you KeePass)
My initial thought was to figure out which boxes currently interact with the RAC boxes, and then regenerate the keys per [note 300548.1|300548.1]. Now, I've only been working with RAC for 6 months since I switched jobs, so while I'm getting more comfortable with the tech, there's still a lot I haven't explored thus far, and we don't have a test RAC environment for me to just start trying things (it's in the pipe). The other DBA here has more experience with RAC, but hasn't actually set one up, so neither of us knows for sure what the full ramifications of resetting the keys might be. Could this effect CRS negatively or is it using its own connections through ports on the private network addresses? Aside from non-clustered boxes needing their keys updated, are there any other considerations we should be thinking about?
Thanks!

"User equivalence unavailable on all the nodes"After install Oracle RAC and you create phrase key (have password), you'll find it. You can ignore after install Cluster completely
But when you use "cluvfy" or etc... about RAC (dbca/netca)
you have to
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
on metalink 300548.1, they try to told us... create key with no password
$ ssh-keygen -t rsa
Now accept the default location for the key file
Enter and confirm a passphrase. (you can also press enter twice).
$ ssh-keygen -t dsa
Now accept the default location for the key file
Enter and confirm a passphrase. (you can also press enter twice).
So, When we use cluvfy" or etc... (with creating key with no password), we don't run "ssh-agent $SHELL" ...
and I think You'll not find "User equivalence unavailable on all the nodes" as well.

Manually Setting up a Standby Database for RAC Environment

Hi,
I am a Junior DBA so apologies if this sounds like a silly question.
We have production 11g Extended RAC environment setup. Management have asked for a separate single instance database be setup as a standby DB on a separate 'cold' server in case we lose complete connection to both sides of the RAC. I read up on Oracle Data Guard and presented this as a solution, but they seem adamant about manually going in and copying out the latest backup and archive logs over to the standby database.
Can this be done? I mean, can ASM managed backup files and archive logs from one database be simply copied out of the backup directory and imported into a completely separate database as easy as that?
We are using ASM to manage to data files on the RAC. My understanding is that we cant manually access the files on an OS level when using ASM, but maybe I am wrong. Any help or opinions on this would be greatly appreciated.
Rgs,
Rob

Hi,
Can this be done? I mean, can ASM managed backup files and archive logs from one database be simply copied out of the backup directory and imported into a completely separate >database as easy as that?Yes, but depends on DB version.
what is the DB version.
from 11g you have cp command so there is a possibility to copy generated archive logs to standby location, so that you can apply..
In 10g there is no such option. you need to create a standby database. with automatic shipping
Thanks

More on Extended RAC

Similar Messages

Maybe you are looking for