Dedicated switches needed for RAC interconnect or not?

Currently working on an Extended RAC cluster design implementation, I asked the network engineer for dedicated switches for the RAC interconnects.
Here is a little background:
There are 28 RAC clusters over 2X13 physical RAC nodes with separate Oracle_Home for each instance with atleast 2+ instances on each RAC node. So 13 RAC nodes will be in each site(Data-Center). This is basically an Extended RAC solution for SAP databases on RHEL 6 using ASM and Clusterware for Oracle 11gR2. The RAC nodes are Blades in a c7000 enclosure (in each site). The distance between the sites is 55+ kms.
Oracle recommends to have Infiniband(20GBps) as the network backbone, but here DWDM will be used with 2X10 Gbps (each at 10 GBps) links for the RAC interconnect between the sites. There will be separate 2x1GBps redundant link for the Production network and 2x2 GBps FC(Fiber-Channel) redundant links for the SAN/Storage(ASM traffic will go here) network. There will be switches for the Public-production network and the SAN network each.
Oracle recommends dedicated switches(which will give acceptable latency/bandwith) with switch redundancy to route the dedicated/non-routable VLANs for the RAC interconnect (private/heartbeat/global cache transfer) network. Since the DWDM interlinks is 2x10Gbps - do I still need the dedicated switches?
If yes, then how many?
Your inputs will be greatly appreciated.. and help me take a decision.
Many Thanks in advance..
Abhijit

Absolutely agree.. the chances of overload in a HA(RAC) solution and ultmate RAC node eviction are very high(with very high latency) and for exactly this reason I even suggested inexpensive switches to route the VLANs for the RAC interconnect through these switches. The ASM traffic will get routed through the 2x2GB FC links through SAN-Directors (1 in each site).
Suggested the network folks to use Up-links from the c7000 enclosure and route the RAC VLAN through these inexpensive switches for the interconnect traffic. We have another challenge here: HP has certified using VirtualConnect/Flex-Fabric architecture for Blades in c7000 to allocate VLANs for RAC interconnect. But this is only for one site, and does not span Production/DR sites separated over a distance.
Btw, do you have any standard switch model to select from.. and how many to go for a RAC configuration of 13 Extended RAC clusters with each cluster hosting 2+ RAC instances to host total of 28 SAP instances.
Many Thanks again!
Abhijit

Similar Messages

Is Veritas- or Sun Cluster needed for RAC in a Solaris Environment

Is a Veritas- or Sun Cluster needed for RAC in a Solaris Environment?
Does anyone know, when OCFS will be available for Solaris?

You don't need Veritas Cluster File System, but until OCFS comes out for Solaris you need to think about backups. If you've not got a backup solution that can integrate with rman for an SBT device then backups become more tricky.
If you use ASM then you can take a backup to a "cluster filesystem" (although ASM is raw partitions think of it as a cluster filesystem), that both nodes can see. BUT you then need to get these to tape somehow, unless you've got NetBackup et al. that support RMAN and can backup direct to tape you're more stuck.
Too many people don't think about this. You could create an NFS mount and backup to this from the nodes.

Teamed NICs for RAC interconnect

Hi there,
We have an Oralce 10g RAC with 2 nodes. there are only one NIC for RAC interconnect in both servers.
now we want to add one redundant NIC into each server for RAC interconnect as well.
Could you please guide me some documents about this "teamed NICs for RAC interconnect "?
Your help is greatly appreciated!
Thanks,
Scott

Search around for NIC bonding. The exact process will depend on your OS.
Linux, see Metalink note 298891.1 - Configuring Linux for the Oracle 10g VIP or private interconnect using bonding driver
Regards,
Greg Rahn
http://structureddata.org

Hi! I can't upgrade my iTunes 10.3.1.55 on my Windows XP 2002 SP3 to the latest version of iTunes. Got the message: "A problem has occured with the Windows Installer-package. A program needed for this installation could not be run." What to do?

Hi! I can't upgrade my iTunes 10.3.1.55 on my Windows XP 2002 SP3 to the latest version of iTunes. Got the message: "A problem has occured with the Windows Installer-package. A program needed for this installation could not be run." What to do?

Perhaps let's first try updating your Apple Software Update.
Launch Apple Software Update ("Start > All Programs > Apple Software Update"). Does it launch and offer you a newer version of Apple Software Update? If so, choose to install just that update to Apple Software Update. (Deselect any other software offered at the same time.)
If the ASU update goes through okay, try another iTunes install. Does it go through without the errors this time?

In-place upgrade of Enterprise Manager for RAC databases is not supported

upgrading 2 node RAC from 10.2.0.1 to 10.2.0.3 on Windows 2003 64 bit. I got the following:
In-place upgrade of Enterprise Manager for RAC databases is not supported in this release
Now I cannot log in to the database control as sys with sysdba role . I get "invalid username/password" error
I can login as a dba user
I also get a java error after I log in, some thing about class not found, but then I'm in and I can do everything that I try, although I haven't tried scheduling a backup yet
Can I upgrade EM to 10.2.0.3?
If so, how?

On the patchset readme...
11.3 Upgrade of Oracle Enterprise Manager Not Supported
When you start Oracle Database Upgrade Assistant to upgrade 10.2.0.1 database to 10.2.0.3 patch set, the following error occurs:
Inplace upgrade of Enterprise Manager is not supported for RAC databases is not supported in this release
Workaround:
Ignore warning.
If you drop and recreate the EM repository?
1). emca -deconfig dbcontrol db repos drop
2). emca -config dbcontrol db -repos create
Be careful, doing this put the database on QUIESCE mode.
Regards,
Rodrigo Mufalani

802.3ad (mode=4) bonding for RAC interconnects

Is anyone using 802.3ad (mode=4) bonding for their RAC interconnects? We have five Dell R710 RAC nodes and we're trying to use the four onboard Broadcom NetXtreme II NICs in a 802.3ad bond with src-dst-mac load balancing. Since we have the hardware to pull this off we thought we'd give it a try and achieve some extra bandwith for the interconnect rather than deploying the traditional acitve/standby interconnect using just two of the NICs. Has anyone tried this config and what was the outcome? Thanks.

I don't but may be the documents might help ?
http://www.iop.org/EJ/article/1742-6596/119/4/042015/jpconf8_119_042015.pdf?request-id=bcddc94d-7727-4a8a-8201-4d1b837a1eac
http://www.oracleracsig.org/pls/apex/Z?p_url=RAC_SIG.download_my_file?p_file=1002938&p_id=1002938&p_cat=documents&p_user=nobody&p_company=994323795175833
http://www.oracle.com/technology/global/cn/events/download/ccb/10g_rac_bp_en.pdf
Edited by: Hub on Nov 18, 2009 10:10 AM

Why can't I have the Samsung Haven replaced without paying for a new phone since the battery needed for this model will not hold a charge more then 24 hrs?

The Haven is my husbands phone and he calls me from work every night and we talk about 15 minutes. He calls my verizon phone so there is no charge but the problem we have is the phone will not stay charged. We took it in and had it tested along with the house charge and the car charge. All three device are fine so they ordered us a new battey. We put that in and the same thing happens. We don't text, receive text, take or send pictures. We have texting blocked so nothing can use up his battery and yet it still dies. Can you please tell me how we can get this phone replaced without buying a new one. The contract isn't up for renewal until Oct. 2013. We have been long time customers of verizon and I feel we should be able to get a product replaced if it is a defective product the company is selling. It seems Verizon knows there is a problem since there have been so many complaint filed in the last 9 months. If we buy something from Home Depot or Lowes and that product is not working right, they replace it and they go back to the companies that supply it. I am willing to bet Samsung would take Verizon,s complaint more seriously then they would an average consummer. Please tell me Verizon is finally listening to their customers.

Hi THerman!
We definitely don't want you having a phone described as a lemon so let's work on getting everything back up to a peachy status. What you've indicated is certainly an issue. So, let's fix this.
First, when it comes to your calls not ringing or registering in your log, my recommendation is to update your phone. To do this, simply dial *228, choose option 2, wait for completion and for your phone to restart itself. This will have it resync with our network for optimal usage. Regarding the battery, personally I like to pull my battery every now and again for up to 60 seconds before reinserting (all while the phone is powered off). If you don't notice improvement, we may need to look into a replacement if life is shortlived below what was described prior.
EvanO_VZW
VZW Support
Follow Us On Twitter @VZWSupport

Help needed for importing Word with notes

Hello everyone,
I have a Word document that has about 80 short sections to it. Each section has endnotes, and the note numbering restarts with 1 at the beginning of each section. I have tried to import this document into CS3, but each time the notes appear as endnotes at the end of the document, not the end of each section. Worse, they number consecutively across sections. Quark, which also dumps notes at the end of the document, at least retains their original numbering, so I could cut-and-paste them back in place. Any information or suggestions on how to work this in INDD? Many thanks.

mgladue wrote:
Hello everyone,
I have a Word document that has about 80 short sections to it. Each section has endnotes, and the note numbering restarts with 1 at the beginning of each section. I have tried to import this document into CS3, but each time the notes appear as endnotes at the end of the document, not the end of each section. Worse, they number consecutively across sections. Quark, which also dumps notes at the end of the document, at least retains their original numbering, so I could cut-and-paste them back in place. Any information or suggestions on how to work this in INDD? Many thanks.
This Google search should find at least a few useful suggestions: InDesign word endnotes side headings script kahrel. "Kahrel" is for Peter Kahrel, who's contributed many useful InDesign scripts to the community, especially one that manipulates endnotes.
HTH
Regards,
Peter
Peter Gold
KnowHow ProServices

RAC - Interconnect traffic

In the client place the architecture team wants to implement a a node RAC cluster on Sun Solaris on Oracle 10g. But in order to minimize the interconnect traffic they want applications to connect only to one node and the other node will provide fail over capability.
Though this can be achieved with less complexity thru physical standby, theoretically if in a 2 node RAC cluster if we just use one node will it eliminate interconnect traffic?
I guess that there will not be any requests from the second node for blocks held by the first node so no transfer of blocks over interconnect will take place. But still the second node has to know about the blocks held by the first node in order to recover in case the first node crashes(cache fusion?). Right?
TIA
RadKrish

just my 0.02 cents on this .. Why would you want to invest in a High Availability Solution ( RAC ) and attempt to achieve Disaster Recovery ( Failover ) through it ..
Why would you want to minimize the Interconnect traffic in the first place .. A decent Switch plus Gigabit Ethernet / Cat5 cables is all you need for the Interconnect and its known to work well with most types of RAC Setup's ..
With Dynamic Resource Remastering , mastership of blocks across the active nodes get distributed based on access patterns .. so with your kind of setup , almost all the blocks would be mastered on Node:1 after a period of sustained activity .. Further Instance recovery happens using information from the Redo Log Thread of the crashed Node so it should not affect Interconnect traffic ..
Vishwa

RAC Interconnect Transfer rate vs NIC's Bandwidth

Hi Guru,
I need some clarification for RAC interconnect terminology between "private interconnect transfer rate" and "NIC bandwidth".
We have 11gR2 RAC with multiple databases.
So we need to find out what the current resource status is.
We have two physical NICs each node. And 8G is for public and 2G is for private (interconnect).
Technically, we have 4G for Private network bandwidth.
If I look at the "Private Interconnect Transfer rate" though OEM or IPTraf (linux tool), it is showing 20 ~30 MB/Sec.
There is no any issue at all at this moment.
Please correct me if I am wrong.
The transfer rate will be fine till 500M or 1G/Sec. Because the current NIC's capacity is 4G. Does it make sense ?
I'm sure there are multiple things to consider,but I'm kind of stumped on the whole transfer rate vs bandwidth. Is there any way to calculate what a typical transfer would be....
OR How do I say our interconnect are good enough ....based on the transfer rate ?
Another question is ....
In our case, how do I set up the warning threshold and Critical threshold for "Private Interconnect Transer rate" in OEM ?
Any comments will be appreciated.
Please advise.

Interconnect performance sways more to latency than bandwidth IMO. In simplistic terms, memory is shared across the Interconnect. What is important for accessing memory? The size of the pipe? Or the speed of the pipe?
A very fast small pipe will typically perform significantly better than a large and slower pipe.
Even the size of the pipe is not that straight forward. Standard IP MTU size is 1500. You can run jumbo and super-jumbo frame MTU sizes on the Interconnect - where for example a MTU size of 65K is significantly larger than a 1500 byte MTU. Which means significantly more data can be transferred over the Interconnect at a much reduced overhead.
Personally, I would not consider Ethernet (GigE included) for the Interconnect. Infiniband is faster, more scalable, and offers an actual growth path to 128Gb/s and higher.
Oracle also uses Infiniband (QDR/40Gb) for their Exadata Database Machine product's Interconnect. Infiniband also enables one to run Oracle Interconnect over RDS instead of UDP. I've seen Oracle reports to the OFED committee saying that using RDS in comparison with UDP, reduced CPU utilisation by 50% and decreased latency by 50%.
I also do not see the logic of having a faster public network and a slower Interconnect.
IMO there are 2 very fundamental components in RAC that determines what is the speed and performance achievable with that RAC - the speed, performance and scalability of the I/O fabric layer and for the Interconnect layer.
And Exadata btw uses Infiniband for both these critical layers. Not fibre. Not GigE.

Doubts about shared disk for RAC

Hi All,
I am really new to RAC.Even after reading various documents,I still have many doubts regarding shared storage and file systems needed for RAC.
1.Clusterware has to be installed on a shared file system like OCFS2.Which type of hard drive is required to install OCFS2 so that it can be accessed from all nodes??
It has to be an external hard drive???Or we can use any simple hard disk for shared storage??
If we use external hard drive then does it need to be connected to a seperate server alltogether or can it be connected to any one of the nodes in the cluster???
Apart from this shared drives,approximately what size of hard disk is required for all nodes(for just a testing environment).
Sincerely appreciate a reply!!
Thanks in advance.

Clusterware has to be installed on shared storage. RAC also requires shared storage for the database.
Shared storage can be managed via many methods.
1. Some sites using Linux or UNIX-based OSes choose to use RAW disk devices. This method is not frequently used due to the unpleasant management overhead and long-term manageability for RAW devices.
2. Many sites use cluster filesystems. On Linux and Windows, Oracle offers OCFS2 as one (free) cluster filesystem. Other vendors also offer add-on products for some OSes that provide supported cluster filesystems (like GFS, GPFS, VxFS, and others). Supported cluster filesystems may be used for Clusterware files (OCR and voting disks) as well as database files. Check Metalink for a list of supported cluster filesystems.
3. ASM can be used to manage shared storage used for database files. Unfortunately, due to architecture decisions made by Oracle, ASM cannot currently be used for Clusterware files (OCR and voting disks). It is relatively common to see ASM used for DB files and either RAW or a cluster filesystem used for Clusterware files. In other words, ASM and cluster filesystems and RAW are not mutually exclusive.
As for hardware--I have not seen any hardware capable of easily connecting multiple servers to internal storage. So, shared storage is always (in my experience) housed externally. You can find some articles on OTN and other sites (search Google for them) that use firewire drives or a third computer running openfiler to provide the shared storage in test environments. In production environments, SAN devices are commonly employed to provide concurrent access to storage from multiple servers.
Hope this helps!
Message was edited by:
Dan_Norris

Is that true for RAC vs Redhat on cost?

Hi,
see below link, video,
http://www.redhat.com/promo/streamline/HighAvailabilityDatabase/?intcmp=70160000000HUrE
So Redhat has better RAC solution.
Redhat RAC + Oracle Standard edtion, beat Oracle enterprise? performance, and cost.
Peter

> Is that true for RAC vs Redhat on cost?
No. The assumption from Redhat in this video is that the application and data can be neatly partitioned into customers and orders. (a shared nothing cluster approach)
If this is truly the case (which will be unusual), then there is really very little need for RAC - and many other solutions that are even better than this "offering" from Redhat. (exactly what are they trying sell here anyway?)
But as this neat and tidy border very seldom exist, how do you integrate the separated data sets? A few methods.
You can look at it from the db side. Distributed selects over database links (that can easily kill performance). Materialised views (more expensive SAN space). Etc.
You can look at it from the app side. Make the app use dual connections, one to each database. Move the joining logic and code from the database into the app. Of course, a great way to turn the app fat, make the database a bit bucket, and kick performance repeatedly in the nuts.
You can look at it from the app server side. Have the app server (with its scalability etc) access the databases, integrate the data and cache the data. Only.. isn't this exactly what a database does? And what about the cost for these app servers that needs tons of memory for caching. (Hint to Redhat: JBoss ain't that good and makes a poor database)
Is RAC expensive? That depends... If you go and buy huge SMP boxes as RAC nodes, your h/w is going to be as expensive as your s/w. If you buy MPP blades, your h/w cost will be a fraction of your s/w cost.
And that is a key point that is often missed. And one that I had all kinds of arguments with management about. There is a complexity factor in doing high availability, redundancy and scalability.
With MPP that complexity moves from the h/w into the s/w. So unlike the traditional approach (dating from buying very very expensive mainframe h/w), you buy cheap h/w. And as the complexity is in the s/w, you pay more for the s/w. Google's cluster of clusters is the prime example of this.
Thus the cost moves from the h/w to the s/w. So is RAC really that expensive? And remember that one needs to compare apples to apples here. If you price another solution to compare with RAC, that solution needs to provide the same feature set you want from RAC.
And that's where the Redhat video falls flat. They do not offer the same feature set. And if that is truly the feature set needed by the client, then RAC is the wrong architecture to use in the first place.

RAC interconnect switch?

Hi,
We are in the process of migrating from a 10g single instance database to 2 node RAC (Windows Server 2008 OS, EMC storage with 2 SAN swithes,…) and we have some doubts about interconnect.
We are having difficulty in selecting the correct interconnect speed for the interconnect network, difficulty in selecting the switch/switches, …
1.     Because there are 2 nodes, and 4 Ethernet cables for interconnect, whether to use one or two switches? Using a switch can be a solution but a single switch become a big single point of failure.
2.     Whether we can get in performance if we use 2 switches (bonding,…) ?
3.     As mentioned, there are 4 Ethernet cables, is it good idea to use existing 1Gb switches that we use for public network or to buy 1Gb switches that will be used only for private interconnection?
4.     Can we use simple 16 or 8 port GigE switches?
Maybe you can point me out to some GigE and SAN switches (for nodes - storage connection) which you've seen that they work without any problems with RAC.
How can we best deisgn the networks for the interconnects?
Thank in advance!

user9106065 wrote:
So the best solution for interconnection would be InfiniBand or 10GigE.If you look at what Oracle itself chose for their RAC hardware product range, then yes. Infiniband is a better choice.
What do you think about InfiniBand and Windows Server OS?Last used Windows as a server o/s back in the mid 90's. :-)
No idea how robust the OFED driver stack is on Windows. It ships with Oracle Linux as Oracle uses it for their RAC products.
What is the difference in price for InfiniBand and GigE switches?About the same I would think. A 40Gb 12 to 24 port switches a few years ago were actually cheaper than a 10GigE switch of the same port count. Pricing has come down for both though. We have recently bought a couple of 32 port QDR switches at far below $10,000 a switch.
Cabling is needed and HCA (PCI) cards too. The cards are cheaper I think than HBAs (fibre channel cards). The only issue we had in this regard is getting pizza box/blade servers with 2 PCI slots for supporting both HBA and HCA. Recent server models often have only one PCI slot as oppose to the prior models of a few years ago. So when choosing a newer servers and you need both HBA and HCA, you need to make sure there are in fact 2 PCI slots in the server.
And again, can you point me out to some InfiniBand switches which you've seen that they work without any problems with 2 node RAC?Oracle used Voltaire IB (Infiniband) switches for the first Oracle Database Machine/Exadata product. The only top500 cluster in Africa is basically (almost) next door to us here in Cape Town. They are also using Voltaire switches.
If I'm not mistaken, the same Voltaire switches are OEM'ed and sold by Oracle/Sun and HP and others. I have an HP quote for about the same below $10,000 per switch price. Of course, you can get away with a much smaller switch for a 2 node RAC - and a 2nd switch is only a consideration if you can justify the cost for redundancy in the Interconnect redundancy layer.
Voltaire pretty much seems to lead the market in this respect. Cisco used to sell IB switches too - but some of these were horribly buggy (especially the ones with FC gateways). Cisco acquired TopSpin back then - we still have a couple of old 10Gb TopSpin switches (bought from Cisco) and these have been pretty rock solid through the years. But QDR (40Gb) is what one should be looking at and not the older SDR or DDR technologies.
You should be able to shop around your existing vendors (HP, Oracle/Sun, etc) for IB switches - with the exception of Cisco that no longer does IB switches (afaik).

Su[12345] : passwdutil.so: nameservice switch entry for passwd not found.

Hello all,
I got a problem with my Sun Fire, with solaris 10, when my system boot, we got a powerfailed :-( and when my sun restart again after a long checkdisk with many errors... i got this error everytime i want to do a su.
su[12345] : passwdutil.so: nameservice switch entry for passwd not found.
I don't know how to fix that.
If anyone have a working solution, i would be wonderfull :)
Thanks in advance

Use your service contract and open a support case with Sun.
http://www.sun.com/secure/contact/
You will then be able to draw upon all their support resources.
They will be able to help you investigate the health of all the hardware,
as well as work with you to decide whether you need to install all your
software all over again, due to too much corruption in the filesystems.
Electrical interruptions can sometime do some nasty things to data on disk drives.

I buy for mac app but not for the iPad, so how do I reinstall a program that has already purchased? I need help because I bought the imovie'11 but when he opened it only brought 8 movie trailer templates instead of the 15 it says on the program. How do I

I buy for mac app but not for the iPad, so how do I reinstall a program that has already purchased? I need help because I bought the imovie'11 but when he opened it only brought 8 movie trailer templates instead of the 15 it says on the program. How do I solve this problem? Once paid for a program incomplete. I also liked that the AppStore itself had a button to reinstall programs already bought, because it is very hard to understand how to do this in the store, only option is to install and hide purchase, and how do I reinstall, attention'm talking about reinstalling a computer mac, not an ipad or iphone.
Thank you.

Prodesigntools.com , go ahead and download the software from there and use your serial number to serialize it.

Dedicated switches needed for RAC interconnect or not?

Similar Messages

Maybe you are looking for