RCA for Oracle RAC Performance Issue

Hi DBAs,
I have setup a 2 node Oracle RAC 10.2.0.3 on Linux 4.5 (64 bit) with 16 GB memory and 4 dual core CPUs each. The database is serving a web application but unfortunately the system is at its knees. The performance is terrible. The storage is a EMC SAN but ASM is not implemented with a fear to further degrade the performance or not to complicate the system further.
I am seeking the expert advises from some GURUs from this forums to formulate the action plan to do the root cause analysis to the system and database. Please advise me what tools I can use to gather the information about the Root Cause. AWR Report is not very helpful. The system stats with top, vmstat, iostat only show the high resource usage but difficult to find the reason. OEM has configured and very frequently report all kind of high wait events.
How I can use effectively find Network bottle necks (netstat command which need to be really helpful to understand).
How I can see the system I/O (iostats) which can provide me some useful information. I don't understand what sould be the baseline or optimal values to compare the I/O activities.
I am seeking help and advised to diagnose the issue. I also want to represent this issue as a case study.
Thanks
-Samar-

First of all, RAC is mainly suited for OLTP applications.
Secondly, if your application is unscalable (it doesn't use bind variables and no SQL statements have been tuned and/or it has been ported from Sukkelserver 200<whatever>) running it against RAC will make things worse.
Thirdly: RAC uses a chatty Interconnect. If you didn't configure the Interconnect properly,and/or are using slow Network cards (1 Gb is mandatory), and/or you are not using a 9k MTU on your 1 Gb NIC, this again will make things worse.
You can't install RAC 'out of the box'. It won't perform! PERIOD.
Fourthly: you might suffer from your 'application' connecting and disconnecting for every individual SQL statement and/or commit every individual INSERT or UPDATE.
You need to address this.
Using ADDM and/or AWR is compulsory for analysing the problem, and/or having read Cary Millsaps book on Optimizing Oracle performance is compulsory.
You won't come anywhere without AWR and OS statistics will not provide any clue.
Because, paraphrasing William Jefferson Clinton, former president of the US of A:
It's the application, stupid.
99 out of 100 cases. Trust me. All developers I know currently are 100 percent clueless.
That said, if you can't be bothered to post the top 5 AWR events, and you aren't up to using AWR reports, maybe you should hire a consultant who can.
Regards,
Sybrand Bakker
Senior Oracle DBA

Similar Messages

Gig Ethernet V/S SCI as Cluster Private Interconnect for Oracle RAC

Hello Gurus
Can any one pls confirm if it's possible to configure 2 or more Gigabit Ethernet interconnects ( Sun Cluster 3.1 Private Interconnects) on a E6900 cluster ?
It's for a High Availability requirement of Oracle 9i RAC. i need to know ,
1) can i use gigabit ethernet as Private cluster interconnect for Deploying Oracle RAC on E6900 ?
2) What is the recommended Private Cluster Interconnect for Oracle RAC ? GiG ethernet or SCI with RSM ?
3) How about the scenarios where one can have say 3 X Gig Ethernet V/S 2 X SCI , as their cluster's Private Interconnects ?
4) How the Interconnect traffic gets distributed amongest the multiple GigaBit ethernet Interconnects ( For oracle RAC) , & is anything required to be done at oracle Rac Level to enable Oracle to recognise that there are multiple interconnect cards it needs to start utilizing all of the GigaBit ethernet Interfaces for transfering packets ?
5) what would happen to Oracle RAC if one of the Gigabit ethernet private interconnects fails
Have tried searching for this info but could not locate any doc that can precisely clarify these doubts that i have .........
thanks for the patience
Regards,
Nilesh

Answers inline...
Tim
Can any one pls confirm if it's possible to configure
2 or more Gigabit Ethernet interconnects ( Sun
Cluster 3.1 Private Interconnects) on a E6900
cluster ?Yes, absolutely. You can configure up to 6 NICs for the private networks. Traffic is automatically striped across them if you specify clprivnet0 to Oracle RAC (9i or 10g). That is TCP connections and UDP messages.
It's for a High Availability requirement of Oracle
9i RAC. i need to know ,
1) can i use gigabit ethernet as Private cluster
interconnect for Deploying Oracle RAC on E6900 ? Yes, definitely.
2) What is the recommended Private Cluster
Interconnect for Oracle RAC ? GiG ethernet or SCI
with RSM ? SCI is or is in the process of being EOL'ed. Gigabit is usually sufficient. Longer term you may want to consider Infiniband or 10 Gigabit ethernet with RDS.
3) How about the scenarios where one can have say 3 X
Gig Ethernet V/S 2 X SCI , as their cluster's
Private Interconnects ? I would still go for 3 x GbE because it is usually cheaper and will probably work just as well. The latency and bandwidth differences are often masked by the performance of the software higher up the stack. In short, unless you tuned the heck out of your application and just about everything else, don't worry too much about the difference between GbE and SCI.
4) How the Interconnect traffic gets distributed
amongest the multiple GigaBit ethernet Interconnects
( For oracle RAC) , & is anything required to be done
at oracle Rac Level to enable Oracle to recognise
that there are multiple interconnect cards it needs
to start utilizing all of the GigaBit ethernet
Interfaces for transfering packets ?You don't need to do anything at the Oracle level. That's the beauty of using Oracle RAC with Sun Cluster as opposed to RAC on its own. The striping takes place automatically and transparently behind the scenes.
5) what would happen to Oracle RAC if one of the
Gigabit ethernet private interconnects fails It's completely transparent. Oracle will never see the failure.
Have tried searching for this info but could not
locate any doc that can precisely clarify these
doubts that i have .........This is all covered in a paper that I have just completed and should be published after Christmas. Unfortunately, I cannot give out the paper yet.
thanks for the patience
Regards,
Nilesh

Any Oracle best practice/standards for inter-DataCente links for Oracle RAC

Hello Oracle Experts,
Am working for a customer to set up Oracle RAC architecture hosting SAP/Non-SAP applications per SLA levels(MC/BC/Standard) specs. Currently my network team needs calculation to arrive at whether we will go for a (1), (2) or (3) 10Gig links for inter DC (Data-Center) for Oracle RAC.. below is additional background:
•     Porting all client SAP/Non-SAP Oracle databases to new 2 data-centers.
•     There will be 10 blades (4x BL680s and 6x BL460s) in each DC (can scale-up/out later on).
•     Clusters architecture to support Extended/Stretched RAC cluster feature
•     Clusters 2-node each(1-datacenter1, 1-datacenter2) and nodes distributed across 2 x c7000 such that no cluster has more than one node in an enclosure.
•     Each node will have - 4 NIC ports ( 2 x public and 2 x private) , 2 dual-port HBA
•     Oracle ASM/ACFS (ASM Cluster File System), Voting Disk, OCR and Database files
•     the versions are Oracle 11g RAC, Oracle 10g RAC and Oracle 9i (for DataGuard/Standby) on RHEL 6 on Proliant Blades (x86) + BladeMatrix
My network colleagues considering using DWDM across the 2 DCs(given the lesser cost?). Am still looking around if there are any Oracle/industry-best practices around this and having a calculation to support that..
Many Thanks in advance..
Regards,
Abhijit

Hi ,
There are no specific set of steps / practices for batch loading contents to ucm . It would be very much dependent on how many contents does the user have to load to UCM and how well the server is configured in terms of performance .
You can get more details from the following documentation link : http://docs.oracle.com/cd/E21043_01/doc.1111/e10792/c02_settings009.htm
Thanks,
Srinath

OIM 9.1.0.2 - Weblogic JDBC Multi Data Sources for Oracle RAC

Does OIM OIM 9.1.0.2 BP07 support Weblogic JDBC Multi Data Sources (Services>JDBC>Multi Data Sources) for Oracle RAC instead of inserting the "Oracle RAC JDBC URL" on JDBC Data Sources for xlDS and xlXADS (Services>JDBC>Data Sources> xlDS|xlXADS > Connection Poll> URL) ?
If yes, is there are any other modifications that need to be made on OIM, or just changing the data sources?

Yes, it's supported. You install against one instance directly of the Rac Server. Then you update the config.xml file and the jdbc resource in your weblogic server with the full rac address. It is documented for installation against RAC. http://docs.oracle.com/cd/E14049_01/doc.9101/e14047/database.htm#insertedID2
-Kevin

Encountered ora-29701 during Sun Cluster for Oracle RAC 9.2.0.7 startup (UR

Hi all,
Need some help from all out there
In our Sun Cluster 3.1 Data Service for Oracle RAC 9.2.0.7 (Solaris 9) configuration, my team had encountered
ora-29701 *Unable to connect to Cluster Manager*
during the startup of the Oracle RAC database instances on the Oracle RAC Server resources.
We tried the attached workaround by Oracle. This workaround works well for the 1^st time but it doesnt work anymore when the server is rebooted.
Kindly help me to check whether anyone encounter the same problem as the above and able to resolve. Thanks.
Bug No. 4262155
Filed 25-MAR-2005 Updated 11-APR-2005
Product Oracle Server - Enterprise Edition Product Version 9.2.0.6.0
Platform Linux x86
Platform Version 2.4.21-9.0.1
Database Version 9.2.0.6.0
Affects Platforms Port-Specific
Severity Severe Loss of Service
Status Not a Bug. To Filer
Base Bug N/A
Fixed in Product Version No Data
Problem statement:
ORA-29701 DURING DATABASE CREATION AFTER APPLYING 9.2.0.6 PATCHSET
*** 03/25/05 07:32 am ***
TAR:
PROBLEM:
Customer applied 9.2.0.6 patchset over 9.2.0.4 patchset.
While creating the database, customer receives following error:
ORA-29701: unable to connect to Cluster Manager
However, if customer goes from 9.2.0.4 -> 9.2.0.5 -> 9.2.0.6, the problem does not occur.
DIAGNOSTIC ANALYSIS:
It seems that the problem is with libskgxn9.so shared library.
For 9.2.0.4 -> 9.2.0.5 -> 9.2.0.6, the install log shows the following:
installActions2005-03-22_03-44-42PM.log:,
[libskgxn9.so->%ORACLE_HOME%/lib/libskgxn9.so 7933 plats=1=>[46]langs=1=> en,fr,ar,bn,pt_BR,bg,fr_CA,ca,hr,cs,da,nl,ar_EG,en_GB,et,fi,de,el,iw,hu,is,in, it,ja,ko,es,lv,lt,ms,es_MX,no,pl,pt,ro,ru,zh_CN,sk,sl,es_ES,sv,th,zh_TW, tr,uk,vi]]
installActions2005-03-22_04-13-03PM.log:, [libcmdll.so ->%ORACLE_HOME%/lib/libskgxn9.so 64274 plats=1=>[46] langs=-554696704=>[en]]
For 9.2.0.4 -> 9.2.0.6, install log shows:
installActions2005-03-22_04-13-03PM.log:, [libcmdll.so ->%ORACLE_HOME%/lib/libskgxn9.so 64274 plats=1=>[46] langs=-554696704=>[en]] does not exist.
This means that while patching from 9.2.0.4 -> 9.2.0.5, Installer copies the libcmdll.so library into libskgxn9.so, while patching from 9.2.0.4 -> 9.2.0.6 does not.
ORACM is located in /app/oracle/ORACM which is different than ORACLE_HOME in customer's environment.
WORKAROUND:
Customer is using the following workaround:
cd $ORACLE_HOME/rdbms/lib make -f ins_rdbms.mk rac_on ioracle ipc_udp
RELATED BUGS:
Bug 4169291

Check if following MOS note helps.
Series of ORA-7445 Errors After Applying 9.2.0.7.0 Patchset to 9.2.0.6.0 Database (Doc ID 373375.1)

Copper cable / GigE Copper Interface as Private Interconnect for Oracle RAC

Hello Gurus
Can some one confirm if the copper Cables ( Cat5/RJ45) can be used for Gig Ethernet i.e. Private interconnects for deploying Oracle RAC 9.x or 10gR2 on Solaris 9/10 .
i am planning to use 2 X GigE Interfaces (one port each from X4445 Quad Port Ethernet Adapters) & Planning to connect it using copper cables ( all the documents that i came across is been refering to the fiber cables for Private Interconnects , connecting GigE Interfaces , so i am getting bit confused )
would appretiate if some one can throw some lights on the same.
regards,
Nilesh Naik
thanks

Cat5/RJ45 can be used for Gig Ethernet Private interconnects for Oracle RAC. I would recommend trunking the two or more interconnects for redundancy. The X4445 adapters are compatible with the Sun Trunking 1.3 software (http://www.sun.com/products/networking/ethernet/suntrunking/). If you have servers that support the Nemo framework (bge, e1000g, xge, nge, rge, ixgb), you can use the Solaris 10 trunking software, dladmin.
We have a couple of SUN T2000 servers and are using the onboard GigE ports for the Oracle 10gR2 RAC interconnects. We upgraded the onboard NIC drivers to the e1000g and used the Solaris 10 trunking software. The next update of Solaris will have the e1000g drivers as the default for the SUN T2000 servers.

Why do we use reverse proxy for Oracle RAC Cluster setup

Hello All,
I got this question lately.. "why do we use reverse proxy for Oracle RAC Cluster setup". I know we use the reverse proxy at Middleware level for multiple security reasons.
Thanks..

"why do we use reverse proxy for Oracle RAC Cluster setup".
I wouldn't. I wouldn't use a proxy of any sort for the Cluster Interconnect for sure.
Cheers,
Brian

Cat 6K configuration settings for Oracle RAC

Does anyone have experience with Cat6K design and configuration best practices for Oracle RAC?
TIA
Bill

Bill, personally I have not supported oracle real app clustering, however,I think you may find very good architecture information on this pdf link, is this more or less what you are looking for?
http://www.cisco.com/application/pdf/en/us/guest/netsol/ns50/c649/ccmigration_09186a00807688ce.pdf
you may find other docs here as well.
http://www.cisco.com/en/US/netsol/ns656/networking_solutions_program_home.html
Rgds
Jorge

Considering shared storage for Oracle RAC 10g

Hi, guys!
My Oracle RAC will be run on VMware ESXI 5.5. So, both 2 nodes and shared storage are on VM. Don't blame for this, I dont have another choice.
I am choosing shared storage for Oracle RAC. I am choosing between NFS and ISCSI server, both can be done in RedHat linux or FreeNAS.
Can u, guys, help me to do the choise?
RedHat or FreeNAS
ISCSI or NFS
Any help will be appreciated.

JohnWatson написал(а):
NFS is really easy. Create your zero-filled files, set the ownership and access modes, and point your asm_diskstring at them. Much simpler than configuring an iSCSI target and initiators, and then messing about with ASMlib or udev.
I recorded a public lecture that (if I remember correctly) describes it here, Oracle ASM Free Tutorial
I will be using OCFS2 as cluster FS. Does it make any difference for NFS vs ISCSI?

Need clarification for firmware & MPIO version for Oracle RAC

Dear All,
We have two-node Oracle RAC configuration 10gR2, Windos 2033 R2, 64-Bit OS using ASM and Oracle 11gR1 clusterware on HP Blades. The SAN storage is from IBM. We want to upgrade the firmware and MPIO version as per vendor recommendation.
Is there any requirement for Oracle RAC to have specific version of firmware and MPIO? Is there any matrix available for this? Or is it like both nodes should be able to see same SAN storage (LUN) irrespective of version used?
Thanks

Please note that Oracle RAC doesnt have requirement any sepcific firm ware. If you upgrade the firmware, it should not break any existing config, so pls check with IBM.
Also if you are doing any specific firmware upgrade then IBM should provide the supported version list.
Hope this help,
Regards
Satishbabu Gunukula
http://www.oracleracexpert.com

Adding Box for Oracle RAC(Clustering) to BIW 3.5 Oracle/Linus

Hai Gurus,
Suppose if we have 1 box with Bw3.5 Linux/Oracle and add another box to do Oracle RAC(clustring). Do we need to re-instal the Box 1 also or we can do that with small configuration changes.
Guide me.
Regards,
Mano

Hi Mano
As in most cases there is a short answer and a longer one. Short:
If your system is RAC ready, switching from single instance to cluster database and adding a new node is quite simple. From pure application side there will not be major changes necessary.
Long:
What do i need for oracle RAC, what do i have to do to make my system RAC ready?
Hardware / OS
You will need shared disks, cluster fs, on linux mostly OCFS2. If you don't have it yet, you need to get it and move your database on to it. You would probably need an additional lan interface for the interconnect.
Oracle / Database
You would have to install CRS, that is the oracle cluster software. Your oracle instance, listener will run as services controlled by CRS when RAC is enabled. You will need to configure separate redo threads and undo tablespaces for each node you add. Not least you would probably need an additional costly RAC extension license from your favorite oracle sales rep.
SAP
You should think about separate enqueue server and enqueue replication, although this is not mandatory. This is dependent on your reasons for using RAC. If it is high availability you should go for it.
This is not a complete list, i just wanted to give you an initial idea, please ask if you need further details. If you start into RAC just now, you will have a long way to go, otherwise the short answer will apply.
Best regards
Michael

Hardware for ORACLE RAC.

hi
my company decided to migrate from Oracle Single instance to Oracle RAC . now we have to choice a hardware for it . I want to know which hardware is better for mid-size oracle rac database ? HP4300 (lefthand ) or EVA 4400 . the first one is cheaper and support 10g switches but the second one just support Fibre Channel .
by the way , i want to know that does HP P4300 support oracle rac database ?
thx in adv.
Edited by: user9233061 on Mar 7, 2011 12:59 AM

hi
basically every machine can run Oracle RAC. It's a matter of tuning how well it performs.
Also operating systems implies some issues and demands.
The main HW issue with RAC is shared storage, so if you use disk array that can work with all RAC nodes, you're OK.
You can configure it on direct connections (SAS) or network (iSCSI) or fiber channels. With or w/o redundancy.
Go for it.
Ask also the HP consultant for more details and opinions how they view the specific HW from database point of view.

Oracle RAC performance

Hello all,
I have a 3 node oracle RAC. Each server has 2 intel quad core processors. I was asked to extend the RAC to 5 nodes to increase the processing power. I was told that Intel 6 core processors are in the market. So I was looking at the following options and need your input/expertise or suggestions on the following options.
Option 1: Add 2 more nodes with 2 quad core processors to the existing 3 node RAC. so total cores available to process the load will be 40 cores.
Option 2: completely replace all the above 5 nodes(in option 1) with 4 nodes - each with 2 * 6 core processors. total cores available are 48 core.
option 2 will give have 1 less server to maintain.
will the performance on option 2 be better than option 1? Any there any white papers or documentation to help me take a decision as to which option i should go?
Thanks in advance.
Raj

hi
I would also first try to tune the database. However it demands some professional skills. There are many areas of tuning.
For example, if your storage subsystem runs slowly (often a bottleneck), then you will not benefit from and node's scaling!
Upgrading hardware is the last thing to do, as it is costly.
If you have a budget, then run for it, buy new "toys", why not, but make sure you hit the performance issue lest you spent money with no improvement...brr
So start with storage, if disks are old e.g. 10k RPM replace them with new 15k RPM, or SSD even.
Run AWR, see "Top 5 events", find the slowest subsystem, if this is I/O then invest in disks, if CPU then CPU, etc.
Replacing old hardware (scale-up) have this benefit, that you don't have to buy all components and install all software from the scratch (OS, add nodes).
Also, however it is not forbidden, but it's been said by Oracle, nodes should have similar performance.
Next thing is that adding more than 3 nodes to the RAC cluster , probably will not improve the overall performance in a linear way, and is more expensive.
My suggestion is to scale-up after inspecting and finding the bottleneck of the cluster performance.
Hope this helps,

Oracle 9i Performance Issue High Physical Reads

Dear All,
I have Oracle 9i Release 9.2.0.5.0 database under HP Unix, I have run the query and got following output. Can any body just have a look and advise what to do in the following situation? We have performance issues.
Many thanks in advance
Buffer Pool Advisory for DB: DBPR Instance: DBPR End Snap: 902
-> Only rows with estimated physical reads >0 are displayed
Size for Size Buffers for Est Physical Estimated
P Estimate (M) Factr Estimate Read Factor Physical Reads
D 416 .1 51,610 4.27 1,185,670,652
D 832 .2 103,220 2.97 825,437,374
D 1,248 .3 154,830 2.03 563,139,985
D 1,664 .4 206,440 1.49 412,550,232
D 2,080 .5 258,050 1.32 366,745,510
D 2,496 .6 309,660 1.23 340,820,773
D 2,912 .7 361,270 1.14 317,544,771
D 3,328 .8 412,880 1.09 301,680,173
D 3,744 .9 464,490 1.04 288,191,418
D 4,096 1.0 508,160 1.00 276,929,627

Hi,
Actually you didnt give the exact problem statement.
Seems to be your database is I/O bound. Ok, do the following one by one:
1. Identify the FTS queries and try to create the optimal indexes (depending on the disk reads factor!!) on the problem queries.
2. To reduce the 276M physical reads, you need to allocate more memory to db_cache_size. try 8GB (initially) and then depending on the buffer advisery you can increase further if you have more memory on the box.
3. as a Next step , configure KEEP and RECYCLE cache to get the benefits of reduced I/O by multiple pools. Allocate objects to the KEEP/RECYCLE pools.
Thanks,

Oracle RAC performance Suddenly terminates on one of the two node cluster

I have a strange problem that happens frequently from time to time when My M400 Machine which is a part of two node RAC cluster goes down suddenly
I tried so many times to understand what's the cause behind that but when I read the logs there are so many messages related to the Oracle RAC which I don't have any experience or knowledge about so I hope I can find here any one who can explain to me these log messages knowing that they are always the same
Jun 18 08:30:00 kfc-rac1 sendmail[17709]: [ID 702911 mail.crit] My unqualified host name (kfc-rac1) unknown; sleeping for retry
Jun 18 08:31:00 kfc-rac1 sendmail[17709]: [ID 702911 mail.alert] unable to qualify my own domain name (kfc-rac1) -- using short name
Jun 18 11:44:15 kfc-rac1 iscsi: [ID 454097 kern.notice] NOTICE: unrecognized ioctl 0x403
Jun 18 11:44:15 kfc-rac1 scsi: [ID 243001 kern.warning] WARNING: /pseudo/fcp@0 (fcp0):
Jun 18 11:44:15 kfc-rac1 Invalid ioctl opcode = 0x403
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <bin/rac_udlm_monitor_stop> for resource <rac-udlm-rs>, resource group <rac-fw-rg>, node <kfc-rac1
, timeout <300> secondsJun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <bin/rac_framework_monitor_stop> for resource <rac-fw-rs>, resource group <rac-fw-rg>, node <kfc-r
ac1>, timeout <3600> seconds
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <bin/rac_svm_monitor_stop> for resource <rac-svm-rs>, resource group <rac-fw-rg>, node <kfc-rac1>,
timeout <300> seconds
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <scal_dg_monitor_stop> for resource <scal-racdg-rs>, resource group <scal-racdg-rg>, node <kfc-rac
1>, timeout <300> seconds
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <scal_mountpoint_monitor_stop> for resource <racfs-mntpnt-rs>, resource group <racfs-mntpnt-rg>, n
ode <kfc-rac1>, timeout <300> seconds
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <bin/rac_framework_monitor_stop> completed successfully for resource <rac-fw-rs>, resource group <rac-fw-rg>
, node <kfc-rac1>, time used: 0% of timeout <3600 seconds>
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <bin/rac_udlm_monitor_stop> completed successfully for resource <rac-udlm-rs>, resource group <rac-fw-rg>, n
ode <kfc-rac1>, time used: 0% of timeout <300 seconds>
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <bin/rac_svm_monitor_stop> completed successfully for resource <rac-svm-rs>, resource group <rac-fw-rg>, nod
e <kfc-rac1>, time used: 0% of timeout <300 seconds>
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <bin/rac_udlm_stop> for resource <rac-udlm-rs>, resource group <rac-fw-rg>, node <kfc-rac1>, timeo
ut <300> seconds
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <scal_dg_monitor_stop> completed successfully for resource <scal-racdg-rs>, resource group <scal-racdg-rg>,
node <kfc-rac1>, time used: 0% of timeout <300 seconds>
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <scal_mountpoint_monitor_stop> completed successfully for resource <racfs-mntpnt-rs>, resource group <racfs-
mntpnt-rg>, node <kfc-rac1>, time used: 0% of timeout <300 seconds>
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <scal_mountpoint_postnet_stop> for resource <racfs-mntpnt-rs>, resource group <racfs-mntpnt-rg>, n
ode <kfc-rac1>, timeout <300> seconds
Jun 18 17:09:41 kfc-rac1 SC[SUNW.rac_udlm.rac_udlm_stop]: [ID 854390 daemon.notice] Resource state of rac-udlm-rs is changed to offline. Note that RAC framework will not be stopped by STOP me
thod.
Jun 18 17:09:41 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <bin/rac_udlm_stop> completed successfully for resource <rac-udlm-rs>, resource group <rac-fw-rg>, node <kfc
-rac1>, time used: 0% of timeout <300 seconds>
Jun 18 17:09:42 kfc-rac1 samfs: [ID 320134 kern.notice] NOTICE: SAM-QFS: racfs: Initiated unmount filesystem: vers 2
Jun 18 17:09:43 kfc-rac1 samfs: [ID 522083 kern.notice] NOTICE: SAM-QFS: racfs: Completed unmount filesystem: vers 2
Jun 18 17:09:43 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <scal_mountpoint_postnet_stop> completed successfully for resource <racfs-mntpnt-rs>, resource group <racfs-
mntpnt-rg>, node <kfc-rac1>, time used: 0% of timeout <300 seconds>
Jun 18 17:09:43 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <scal_dg_postnet_stop> for resource <scal-racdg-rs>, resource group <scal-racdg-rg>, node <kfc-rac
1>, timeout <300> seconds
Jun 18 17:09:43 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <scal_dg_postnet_stop> completed successfully for resource <scal-racdg-rs>, resource group <scal-racdg-rg>,
node <kfc-rac1>, time used: 0% of timeout <300 seconds>
Jun 18 17:09:43 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <bin/rac_svm_stop> for resource <rac-svm-rs>, resource group <rac-fw-rg>, node <kfc-rac1>, timeout
<300> seconds
Jun 18 17:09:43 kfc-rac1 SC[SUNW.rac_svm.rac_svm_stop]: [ID 854390 daemon.notice] Resource state of rac-svm-rs is changed to offline. Note that RAC framework will not be stopped by STOP metho
d.
Jun 18 17:09:43 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <bin/rac_svm_stop> completed successfully for resource <rac-svm-rs>, resource group <rac-fw-rg>, node <kfc-r
ac1>, time used: 0% of timeout <300 seconds>
Jun 18 17:09:43 kfc-rac1 Cluster.RGM.global.rgmd: [ID 224900 daemon.notice] launching method <bin/rac_framework_stop> for resource <rac-fw-rs>, resource group <rac-fw-rg>, node <kfc-rac1>, ti
meout <300> seconds
Jun 18 17:09:43 kfc-rac1 SC[SUNW.rac_framework.rac_framework_stop]: [ID 854390 daemon.notice] Resource state of rac-fw-rs is changed to offline. Note that RAC framework will not be stopped by
STOP method.
Jun 18 17:09:43 kfc-rac1 Cluster.RGM.global.rgmd: [ID 515159 daemon.notice] method <bin/rac_framework_stop> completed successfully for resource <rac-fw-rs>, resource group <rac-fw-rg>, node <
kfc-rac1>, time used: 0% of timeout <300 seconds>
Jun 18 17:09:44 kfc-rac1 root: [ID 702911 user.error] Oracle CRSD 3932 set to stop
Jun 18 17:09:44 kfc-rac1 root: [ID 702911 user.error] Oracle CRSD 3932 shutdown completed
Jun 18 17:09:44 kfc-rac1 root: [ID 702911 user.error] Oracle EVMD set to stop
Jun 18 17:09:44 kfc-rac1 root: [ID 702911 user.error] Oracle CSSD being stopped
Jun 18 17:09:45 kfc-rac1 xntpd[980]: [ID 866926 daemon.notice] xntpd exiting on signal 15
Jun 18 17:09:45 kfc-rac1 ip: [ID 646971 kern.notice] ip_create_dl: hw addr length = 0
Jun 18 17:09:45 kfc-rac1 pppd[516]: [ID 702911 daemon.notice] Connection terminated.
Jun 18 17:09:47 kfc-rac1 pppd[9462]: [ID 860527 daemon.notice] pppd 2.4.0b1 (Sun Microsystems, Inc.) started by root, uid 0
Jun 18 17:09:47 kfc-rac1 pppd[9462]: [ID 702911 daemon.notice] Connect: sppp0 <--> /dev/dm2s0
Jun 18 17:09:47 kfc-rac1 rpc.metamedd: [ID 702911 daemon.error] Terminated
Jun 18 17:09:48 kfc-rac1 inetd[482]: [ID 702911 daemon.warning] inetd_offline method for instance svc:/network/rpc/scrcmd:default is unspecified. Taking default action: kill.
Jun 18 17:09:48 kfc-rac1 inetd[482]: [ID 702911 daemon.warning] inetd_offline method for instance svc:/network/rpc/metacld:default is unspecified. Taking default action: kill.
Jun 18 17:09:49 kfc-rac1 inetd[482]: [ID 702911 daemon.warning] inetd_offline method for instance svc:/network/rpc/scadmd:default is unspecified. Taking default action: kill.
Jun 18 17:09:50 kfc-rac1 pppd[9462]: [ID 702911 daemon.notice] local IP address 192.168.224.2
Jun 18 17:09:50 kfc-rac1 pppd[9462]: [ID 702911 daemon.notice] remote IP address 192.168.224.1
Jun 18 17:09:50 kfc-rac1 cl_eventlogd[1554]: [ID 247336 daemon.error] Going down on signal 15.
Jun 18 17:09:52 kfc-rac1 ip: [ID 372019 kern.error] ipsec_check_inbound_policy: Policy Failure for the incoming packet (not secure); Source 192.168.224.001, Destination 192.168.224.002.
*Jun 18 17:09:56 kfc-rac1 ip: [ID 646971 kern.notice] ip_create_dl: hw addr length = 0*
*Jun 18 17:09:56 kfc-rac1 pppd[9462]: [ID 702911 daemon.notice] Connection terminated.*
*Jun 18 17:09:56 kfc-rac1 Cluster.PNM: [ID 226280 daemon.notice] PNM daemon exiting.*
*Jun 18 17:09:57 kfc-rac1 pseudo: [ID 129642 kern.info] pseudo-device: tod0*
*Jun 18 17:09:57 kfc-rac1 genunix: [ID 936769 kern.info] tod0 is /pseudo/tod@0*
*Jun 18 17:09:57 kfc-rac1 pseudo: [ID 129642 kern.info] pseudo-device: pm0*
*Jun 18 17:09:57 kfc-rac1 genunix: [ID 936769 kern.info] pm0 is /pseudo/pm@0*
*Jun 18 17:09:57 kfc-rac1 rpc.metad: [ID 702911 daemon.error] Terminated*
Jun 18 17:10:01 kfc-rac1 syslogd: going down on signal 15
*Jun 18 17:10:07 kfc-rac1 rpcbind: [ID 564983 daemon.error] rpcbind terminating on signal.*
*Jun 18 17:10:32 kfc-rac1 Cluster.RGM.fed: [ID 831843 daemon.notice] SCSLM thread WARNING pools facility is disabled*
*Jun 18 17:10:40 kfc-rac1 genunix: [ID 672855 kern.notice] syncing file systems...*
*Jun 18 17:10:40 kfc-rac1 genunix: [ID 904073 kern.notice] done*
Jun 19 14:20:12 kfc-rac1 genunix: [ID 540533 kern.notice] ^MSunOS Release 5.10 Version Generic_141444-09 64-bit
Jun 19 14:20:12 kfc-rac1 genunix: [ID 943908 kern.notice] Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved.
Jun 19 14:20:12 kfc-rac1 Use is subject to license terms.
Jun 19 14:20:12 kfc-rac1 genunix: [ID 678236 kern.info] Ethernet address = 0:21:28:2:21:b2
Thanks in advance for all of you
your response is highly appreciated

Hi I have checked the interconnect between the two nodes and it's as follow
ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.1.100.126 netmask ffffff00 broadcast 10.1.100.255
groupname sc_ipmp0
ether 0:14:4f:3a:6c:19
bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 10.1.100.127 netmask ffffff00 broadcast 10.1.100.255
bge0:2: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
inet 10.1.100.140 netmask ffffff00 broadcast 10.1.100.255
bge1: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 6
inet 172.16.0.129 netmask ffffff80 broadcast 172.16.0.255
ether 0:14:4f:3a:6c:1a
nxge0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 10.1.100.128 netmask ffffff00 broadcast 10.1.100.255
groupname sc_ipmp0
ether 0:21:28:d:c9:8e
nxge1: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 5
inet 172.16.1.1 netmask ffffff80 broadcast 172.16.1.127
ether 0:21:28:d:c9:8f
e1000g1: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 4
inet 172.16.1.129 netmask ffffff80 broadcast 172.16.1.255
ether 0:15:17:81:15:c3
clprivnet0: flags=1009843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,PRIVATE,IPv4> mtu 1500 index 7
inet 172.16.4.1 netmask fffffe00 broadcast 172.16.5.255
ether 0:0:0:0:0:1
sppp0: flags=10010008d1<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 8
inet 192.168.224.2 --> 192.168.224.1 netmask ffffff00
ether 0:0:0:0:0:0
root@kfc-rac1 #
and it's direct attached between both nodes interfaces
back to back
and about the status of the hba cards here is it as well
fcinfo hba-port -l
HBA Port WWN: 2100001b3284c042
OS Device Name: /dev/cfg/c1
Manufacturer: QLogic Corp.
Model: 375-3355-02
Firmware Version: 05.01.00
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Serial Number: 0402R00-0844647023
Driver Name: qlc
Driver Version: 20090519-2.31
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b3284c042
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
HBA Port WWN: 2100001b321c462b
OS Device Name: /dev/cfg/c2
Manufacturer: QLogic Corp.
Model: 375-3355-02
Firmware Version: 05.01.00
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Serial Number: 0402R00-0844646557
Driver Name: qlc
Driver Version: 20090519-2.31
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b321c462b
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
HBA Port WWN: 2100001b32934b3c
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 05.01.00
FCode/BIOS Version: BIOS: 2.2; fcode: 2.1; EFI: 2.0;
Serial Number: 0402R00-0947745866
Driver Name: qlc
Driver Version: 20090519-2.31
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b32934b3c
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
HBA Port WWN: 2101001b32b34b3c
OS Device Name: /dev/cfg/c4
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 05.01.00
FCode/BIOS Version: BIOS: 2.2; fcode: 2.1; EFI: 2.0;
Serial Number: 0402R00-0947745866
Driver Name: qlc
Driver Version: 20090519-2.31
Type: unknown
State: offline
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: not established
Node WWN: 2001001b32b34b3c
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
root@kfc-rac1 #
In addition here is the ocssd log file as well
http://www.4shared.com/file/Txl9DqLW/log_25155156.html?
you'll find on the lines for the dates in which this issue happens
look at 2012-06-09
2012-06-18
2012-06-21
you'll see something related to the voting disk
it suddenly becomes unavailable which causes the problem
thanks a lot for your help
I'm waiting for your recommendation
hope these logs gives more look for the problem
Thanks in advance :)

RCA for Oracle RAC Performance Issue

Similar Messages

Maybe you are looking for