Unexpected Swap Usage in Oracle RAC

hi Experts,
we are running 2 node RAC 10gR2(10.2.0.4) on AIX6.1. we have 196G of physical memory in each and SGA is 144 and PGA is 32 n remaing is for OS on each node. we also have configured SWAP space of 20G . we have 2 physical Standby in single instance one is with LGWR and other one is in ARCH mode. now problum is both nodes shwing more than 50% of Swap usage. which is indicating wrong as SGA and PGA advisor is showing adequate settings of both parameters . so can any one tell me what may be the reason for this memory leakage . one more thing i must add here is we recently shift our RAC from other Servers . but source and destination both are of IBM.

I'm not an AIX guy at all, but the first thing that came to my mind is large page support. Have you implemented large pages?
Also, it didn't take much googling at all to find:
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100883
Hope that helps,
-Mark

Similar Messages

How to install Oracle RAC using 11gR2 on Win 7 32 bit from scratch?

Hiii,
I am a fresher and I have a win 7 system 32 bit with oracle 11gR2.
I want to install RAC using two or more servers but I dont know too much about it.
I tried to search on google but all they are giving information in pieces(for win 7) or about Linux(Please tell me why its all Linux everywhere not windows).
Please let me know if there any site where I can find step by step installation from scratch.
What are basic requirements(/software if needed).
Thank you.

Hi,
First of all forget about Windows!! Oracle is mainly designed for Linux systems. Oracle runs as real multi-process on Linux machines whereas on Windows it runs as a single process with multi-thread.
Memory usage - huge pages,single CPU run queue, dnfs, storage systems... etc. lots of concepts exists.
Besides Windows is one of the most poorly designed operating systems, you should use it only to watch cute kitten movies on youtube... Believe me, you wouldn't want to learn Oracle on Windows...
So, install virtualbox and then setup Oracle Enterprise Linux on it as a first step... By the way you have to deal with DNS server for release 11.X, RAC, you may start with 10g for now.
ORACLE-BASE has all the installation steps explained detailly, search for "oracle base rac install" on google.
I also explained on my blog, but it is a little bit advanced installation using ZFS storage simulator. DNS server installation is also explained: http://ilkertaysi.com/2013/11/11/oracle-rac-installation-with-zfs-storage/

RCA for Oracle RAC Performance Issue

Hi DBAs,
I have setup a 2 node Oracle RAC 10.2.0.3 on Linux 4.5 (64 bit) with 16 GB memory and 4 dual core CPUs each. The database is serving a web application but unfortunately the system is at its knees. The performance is terrible. The storage is a EMC SAN but ASM is not implemented with a fear to further degrade the performance or not to complicate the system further.
I am seeking the expert advises from some GURUs from this forums to formulate the action plan to do the root cause analysis to the system and database. Please advise me what tools I can use to gather the information about the Root Cause. AWR Report is not very helpful. The system stats with top, vmstat, iostat only show the high resource usage but difficult to find the reason. OEM has configured and very frequently report all kind of high wait events.
How I can use effectively find Network bottle necks (netstat command which need to be really helpful to understand).
How I can see the system I/O (iostats) which can provide me some useful information. I don't understand what sould be the baseline or optimal values to compare the I/O activities.
I am seeking help and advised to diagnose the issue. I also want to represent this issue as a case study.
Thanks
-Samar-

First of all, RAC is mainly suited for OLTP applications.
Secondly, if your application is unscalable (it doesn't use bind variables and no SQL statements have been tuned and/or it has been ported from Sukkelserver 200<whatever>) running it against RAC will make things worse.
Thirdly: RAC uses a chatty Interconnect. If you didn't configure the Interconnect properly,and/or are using slow Network cards (1 Gb is mandatory), and/or you are not using a 9k MTU on your 1 Gb NIC, this again will make things worse.
You can't install RAC 'out of the box'. It won't perform! PERIOD.
Fourthly: you might suffer from your 'application' connecting and disconnecting for every individual SQL statement and/or commit every individual INSERT or UPDATE.
You need to address this.
Using ADDM and/or AWR is compulsory for analysing the problem, and/or having read Cary Millsaps book on Optimizing Oracle performance is compulsory.
You won't come anywhere without AWR and OS statistics will not provide any clue.
Because, paraphrasing William Jefferson Clinton, former president of the US of A:
It's the application, stupid.
99 out of 100 cases. Trust me. All developers I know currently are 100 percent clueless.
That said, if you can't be bothered to post the top 5 AWR events, and you aren't up to using AWR reports, maybe you should hire a consultant who can.
Regards,
Sybrand Bakker
Senior Oracle DBA

Oracle RAC - Not getting performance(TPS) as we expect on insert/update

Hi All,
We got a problem while executing insert/update and delete queries with Oracle RAC system, we are not getting the TPS as we expected in Oracle RAC. The TPS of Oracle RAC (for insert/update and delete ) is less than as that of
single oracle system.
But while executing select queries, we are getting almost double TPS as that of Single Oracle System.
We have done server side and client side load balancing.
Can anyone knows to solve this strange behaviour? Shall we need to perform any other settings in ASM/ Oracle Nodes
for better performance on insert/update and delete queries.
The following is the Oracle RAC configuration
OS & Hardware :Windows 2008 R2 , Core 2 Du0 2.66GHz , 4 GB
Software : Oracle 11g 64 Bit R2 , Oracle Clusterware & ASM , Microsoft iSCSI initiator.
Storage Simulation : Xeon 4GB , 240 GB ,Win 2008 R2, Microsoft iSCSI Traget
Please help me to solve this. We are almost stuck with this situation.
Thanks
Roy

Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ ------------------ ----------------- ----------- -----------
DB time(s): 48.3 0.3 0.26 0.10
DB CPU(s): 0.1 0.0 0.00 0.00
Redo size: 523,787.9 3,158.4
Logical reads: 6,134.6 37.0
Block changes: 3,247.1 19.6
Physical reads: 3.5 0.0
Physical writes: 50.7 0.3
User calls: 497.6 3.0
Parses: 182.0 1.1
Hard parses: 0.1 0.0
W/A MB processed: 0.1 0.0
Logons: 0.1 0.0
Executes: 184.0 1.1
Rollbacks: 0.0 0.0
Transactions: 165.8
Instance Efficiency Indicators
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 93.74 Redo NoWait %: 99.96
Buffer Hit %: 99.99 Optimal W/A Exec %: 100.00
Library Hit %: 100.19 Soft Parse %: 99.96
Execute to Parse %: 1.09 Latch Hit %: 99.63
Parse CPU to Parse Elapsd %: 16.44 % Non-Parse CPU: 84.62
Shared Pool Statistics Begin End
Memory Usage %: 75.89 77.67
% SQL with executions>1: 71.75 69.88
% Memory for SQL w/exec>1: 75.63 71.38

Disable Intel CPU Hyper threading in Oracle RAC environment

Dear Expert,
Does disable CPU hyper threading will have direct impact or maybe unexpected behavior to Oracle RAC database?
Actually the background of this question is we have many received packet queuing on the network layer waiting to be processed. Investigation leads to the processor utilization which shows that many processor are still in idle state or not used when the other processors are busy processing incoming request.
I have visited some oracle note like
CPU Count Consideration for Oracle Parameter Setting When Using Hyper-Threading Technology (Doc ID 289870.1)
Oracle and Hyper-Threading (Doc ID 205089.1)
Using PROCESSOR_GROUP_NAME to bind a database instance to CPUs or NUMA nodes (Doc ID 1585184.1)
These document already give useful information but i wonder if there is any possibilities risk or another consideration of disabling cpu hyper threading.
Please let me know if any other information required.
Thanks.
Best Regards,
Jolly

As far as I know it is set as cpu architecture and mobile chips don't have the option. Not 100% as I've never needed to do it.
Better bet would be to check Intel ark and forums with them I expect.

Solaris x86 with Oracle RAC 10g Enterprise Edition Release 10.2.0.3.0

Hello,
Maybe you can help me (new on RMAN backup) in doing this.
I have configured a single Oracle 10g database to have backup with RMAN with following steps:
1. $ mkdir $ORACLE_BASE/rman_scripts
2. $ mkdir $ORACLE_BASE/logs
3. $ mkdir $ORACLE_BASE/tracking
4. $ mkdir $ORACLE_BASE/c_backup
5. $ sqlplus sys/<password> as sysdba
6. SQL> alter system set db_recovery_file_dest_size = 50G scope=both;
7. SQL> alter system set db_recovery_file_dest='${ ORACLE_BASE}/flash_recovery_ area' scope=both;
8. SQL> alter system set log_archive_dest_10='location= use_db_recovery_file_dest';
9. SQL> shutdown immediate
10. SQL> startup nomount
11. SQL> alter database archivelog;
12. SQL> alter database open;
13. SQL> alter database enable block change tracking using file '${ORACLE_BASE}/tracking/rman_ change_track.f';
14. $ rman target /
15. RMAN> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK
TO '/var/opt/oracle/flash_ recovery_area/ORCL/c_backup/% F';
16. RMAN> CONFIGURE CONTROLFILE AUTOBACKUP ON;
17. RMAN> CONFIGURE BACKUP OPTIMIZATION ON;
18. RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
19. RMAN> exit
I need to configure incremental backup with RMAN on a two node Solaris x86 with Oracle RAC 10g Enterprise Edition Release 10.2.0.3.0 installation.
We also use ASM to store database files, and have Oracle software installed on separate file systems (two Oracle roots for Node1 and Node2).
I have following questions:
1) where to put Flash Recovery Area (FRA)?
I saw recommendations to put FRA on the ASM, is this the best way to do it?
2) Can I put FRA on another file system (not on the ASM) which is available only from Node1? This way I can save space on the ASM.
3) Is it possible/recommended to run RMAN from Node1 only?
Below is the script used to run RMAN on the normal Oracle database (without RAC) which I need to change :
=============================================================================================
2.0 Oracle backup script: /opt/app/oracle/rman_scripts/backup.sh
Use this for daily backups, possiblly as a cron job.
Once a week run this: /opt/app/oracle/rman_scripts/backup.sh FULL
All other days of the week: /opt/app/oracle/rman_scripts/backup.sh INCREMENTAL
Note: You may have to change ORACLE_SID, ORACLE_BASE below to match your database.
=============================================================================================
#!/usr/bin/ksh
ORACLE_SID=orcl
ORACLE_BASE=/opt/app/oracle
ORACLE_HOME=${ORACLE_BASE}/product/10.2.0/db_1
PATH=${ORACLE_HOME}/bin:/usr/bin
LOGDIR=${ORACLE_BASE}/logs
LOGFILE=${LOGDIR}/rman.log
if [[ $# < 1 ]]
then
echo "usage: backup.sh FULL|INCREMENTAL"
exit;
fi
BACKUPTYPE=${1}
full='FULL'
incremental='INCREMENTAL'
if [[ $BACKUPTYPE == $full ]]
then
$ORACLE_HOME/bin/rman target / nocatalog log ${LOGFILE} append << eof
run {
backup database;
SQL 'alter system archive log current';
backup archivelog all;
delete noprompt obsolete;
exit;
eof
echo ''
fi
if [[ $BACKUPTYPE == $incremental ]]
then
$ORACLE_HOME/bin/rman target / nocatalog log ${LOGFILE} append << eof
run {
backup database;
backup incremental level 1 database;
SQL 'alter system archive log current';
backup archivelog all;
delete noprompt obsolete;
exit;
eof
echo ''
fi

Hi [email protected],
Q1) where to put Flash Recovery Area (FRA)?
A1) With RAC: on the shared storage
I saw recommendations to put FRA on the ASM, is this the best way to do it?
If you want your backups to be available for both nodes you have to use shared storage or tape using an mml library.
So if you want to use the FRA for rman backups and the database is on ASM just make ASM the standard for the FRA as well.
Q2) Can I put FRA on another file system (not on the ASM) which is available only from Node1? This way I can save space on the ASM.
A2) Than you cannot recover in case Node1 is down. Best would be to send your storage admin to a training course so he can manage the clustered raw devices needed for ASM.
Q3) Is it possible/recommended to run RMAN from Node1 only?
A3) No see A2.
Regards,
Tycho

How to diagnose large swap usage

Hi,
We have a java application that is connecting to our oracle 10g database. When the application is not connected the swap file usage on the solaris box is about 1GB. But soon as we startup the java application and connect it to oracle the swap usage jumps to a massive 16GB (and stays like that until with shutdown the application - it does not really grow any more)
The application actually looks fine and works without problems - expect the database is hammering the machine with the swap file. can anyone give me some hints at where I can look on the database to try and diagnose what is acutally causing this large swap file creation - in an attempt what is this application doing? At the moment I am confused as to what could actually cause this.
thx.
S.

Hi, I'd use the iosnoop, iotop, rwsnoop and rwtop commands available in dtrace package (eg. http://users.tpg.com.au/adsln4yb/DTrace/dtracevstruss.html). It can tell a lot about i/o operations, also according to swap. But it only applies to the OS, I can't help you with the Oracle...

In oracle rac, If user query a select query and in processing data is fetched but in the duration of fetching the particular node is evicted then how failover to another node internally?

In oracle rac, If user query a select query and in processing data is fetched but in the duration of fetching the particular node is evicted then how failover to another node internally?

The query is re-issued as a flashback query and the client process can continue to fetch from the cursor. This is described in the Net Services Administrators Guide, the section on Transparent Application Failover.

Install Oracle RAC 10g (10.2.0.1) on HP-UX B.11.31 U ia64 failed

Hi All
I am installing Oracle RAC 10g 10.2.0.1 on HP-UX B.11.31 U ia64 but can not complete
hosts file
#Public IPs
10.144.1.111 spgdb01
10.144.1.112 spgdb02
#Private IPs
10.144.2.2 spgdb01p
10.144.2.3 spgdb02p
#Virtual IPs
10.144.1.113 spgdb01v
10.144.1.114 spgdb02v
I do installation with runInstaller without error. It copy and link is ok. When I run root.sh then It cannot complete as following
Checking to see if Oracle CRS stack is already configured
Checking to see if any 9i GSD is up
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle/product/10.2.0' is not owned by root
WARNING: directory '/oracle/product' is not owned by root
WARNING: directory '/oracle' is not owned by root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 0: spgdb01 spgdb01p spgdb01
node 1: spgdb02 spgdb02p spgdb02
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
Now formatting voting device: /ora/crs/votedisk01
waitpid(-1, 0x7fffdf50, WUNTRACED) .................................................................................................... [sleeping]
Now formatting voting device: /oracle/oradata1/crs/votedisk02
Now formatting voting device: /oracle/oradata2/crs/votedisk03
Format of 3 voting devices complete.
Startup will be queued to init within 30 seconds.
====================
I have waited for 10 mins but still not complete
Additionally, log from runInstaller, I got
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2011-04-28_12-13-31AM. Please wait ...-bash-4.2$ Oracle Universal Installer, Version 10.2.0.1.0 Production
Copyright (C) 1999, 2005, Oracle. All rights reserved.
Private Interconnect : null
Private Interconnect : null
Private Interconnect : null
Private Interconnect : null
So, please help me fix this issue
Thank you

I had this problem and resolved it by transporting the file to the installation server with the correct ftp datatype (binary).
On page 54 of the install guide (..Server\Oracle_Business_Intelligence\doc\doc\bi.1013\b31765.pdf) that comes with the installation files, there is an instruction to make sure that any ftp activity is done in binary.
This may not have occured with the license.xml file if you use a tool which offers the "feature" of automatic datatype recognition.
Hope this helps.

Oracle Upgrade from Oracle RAC 9.2.0.6 to Oracle 10.2.0.4

Hi All,
Currently, we are running 4 node Oracle RAC environment with below mentioned configuration.
OS: Sun Solaris 5.9
Hardware: Sun E2900
Oracle Version: 9.2.0.6
Veritas Cluster Server: 4.1
We want to upgrade Oracle version to 10g, and currently analyzing the options to perform this. The current database size is 1TB appx and we want to spend minimum application shutdown time running on this database.
As part of upgrade, we also need to upgrade Veritas Cluster Server from 4.1 to 5.1 to support Oracle 10g. It would be great help,, if someone can pass some guideline to perform this task.
We are currently thinking about piecemeal approach, where we can upgrade each node individually and then put them back to cluster. There are some complexities involved, and its really high risk approach.
Thanks a lot in advance for help
Regards,
Manoj

Orcale 10g RAC requires you to install Oracle Clusterware. Oracle Supports running it along side a 3rd party clustering software. Not sure why you're so anxious to upgrade Veritas Clusterware when it will be trivial on the 10g db hosts.

Oracle rac templates 11g R2 buildcluster.sh error

Hi All,
am facing below error, while creating oracle rac templates. kindly let us know how to resolve below error.
===error=========================
Oracle RAC 11gR2 OneCommand (v1.2) for Oracle VM - (c) 2010-2011 Oracle Corporation
   Cksum: [1170221909 255000 racovm.sh] at Sun Jan 5 04:15:14 EST 2014
   Kernel: 2.6.18-194.0.0.0.3.el5xen (i686) [1 processor(s)] 1700 MB
2014-01-05 04:15:14:[printparams:Time :racnode1] Completed successfully in 4 seconds (0h:00m:04s)
2014-01-05 04:15:14:[setsshora:Start:racnode1] SSH Setup for the Oracle user(s)...
INFO (node:racnode1): Running as oracle: /u01/racovm/ssh/setssh-Linux.sh -s -x -c NO -h nodelist -p ***   (setup on 2 node(s): racnode1 racnode2)
ERROR: Failed to create temporary file /tmp/setssh-cretmpQY3958 on localhost, can not proceed
Exiting...
ERROR (node:racnode1): Failed to configure passwordless SSH for the oracle user
2014-01-05 04:15:17:[setsshora:Time :racnode1] Completed with errors in 3 seconds (0h:00m:03s), status: 1
2014-01-05 04:15:17:[buildcluster:Time :racnode1] Completed with errors in 58 seconds (0h:00m:58s), status: 1
thanks,
Mike.

Try this. It worked for me.
Please keep in mind that you will need wait till each step finishes successfully before move to next one
For Step1 and 2, you can skip node(s) on which you didn't execute root.sh yet.
Step 1: As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force" on all nodes, except the last one.
Step 2: As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode" on last node. This command will zero out OCR and VD disk also.
Step 3: As root, run $GRID_HOME/root.sh on all node one by one

How to configure Oracle RAC for Wiki and Blog

Has anyone done Oracle RAC configuration for Oracle Wiki and Blogs?
Thanks,
Frank

Hi Andreas,
Thanks you for your help. I am trying to implement third party external LDAP authentication for APEX and Forms.
So I started with OID and SSO setup to create external Partner Applications. Some reason my oid and sso web login links are not working. I didn't find any errors. I need some help in finding the problem and direction, I already read docs on web but no proper direction. I appreciate your help.
Thanks

Oracle RAC with ASM install failure

Hi Guy's
I've just rebuilt an Oracle RAC system, i've built this system many times and have the build documented. I usually follow the build doc to the tee and everything works fine. I have now been asked to rebuild the system to go into the production evironment. This is a two server standard RAC setup using ASM. We have a HP MSA500 shared storage setup. I've run the cluster verification tool and all the checks have come out fine. Whilst installing the clusterware i get a failed error message whilst it try's to confiure the Oracle Clusterware Configuration Assistant.
The error message i get is:
Command = C:\Windows\system 32\cmd\C Call E:\Oracle\Product\10.2.0\crs\install\crssetup.config.bat
PROT-1:Failed to intialize OCR Config
STEP 1 Checking status of CRS cluster
STEP 2 Creating directories (E:\Oracle\Product\10.2.0\Crs
STEP 3 Configuring OCR repository
ocr upgrade failed with (-1)
I've done this build many times and never seen this error. From what i've read on the internet it looks like it could be something up with my shared storage but i dont know what.
Any help would be greatly appreciated
Thanks
Lee

try to apply ''dd' command against OCR & Voting disks to ensure that their headers are formatted 100% and contains no data from old installation:
dd if=/dev/zero of=YOUR_OCR_DISK bs=1024 count=10000
dd if=/dev/zero of=YOUR_VOTE_DISK bs=1024 count=10000

ASM instances on 2 node Oracle RAC 10g r2 on Red Hat 4 u1

Hi all
I'm experiencing a problem in configuring diskgroups under +ASM instances on a two node Oracle RAC.
I followed the official guide and also official documents from metalink site, but i'm stuck with the visibility of asm disks.
I created fake disks on nfs with Netapp certified storage binding them to block device with the usual trick "losetup /dev/loopX /nfs/disk1 " ,
run "oracleasm createdisk DISKX /dev/loopX" on one node and
"oracleasm scandisks" on the other one.
With "oracleasm listdisks" i can see the disks at OS level in both nodes , but , when i try to create and mount diskgroup in the ASM instances , on the instance on which i create the diskgroup all is well, but the other one doesn't see the disks at all, and diskgroup mount fails with :
ERROR: no PST quorum in group 1: required 2, found 0
Tue Sep 20 16:22:32 2005
NOTE: cache dismounting group 1/0x6F88595E (DG1)
NOTE: dbwr not being msg'd to dismount
ERROR: diskgroup DG1 was not mounted
any help would be appreciated
thanks a lot.
Antonello

I'm having this same problem. Did you ever find a solution?

How to create a wallet in oracle RAC environment

How to create a wallet in oracle RAC environment.
While running following command "alter system set encryption key identified by "thalesdata4";
I am getting error message "cannot auto create wallet" or "failed to open wallet.
Please suggest correct way to create a wallet in RAC environment.
Thanks
Sudhir

hi,
please refer for detailed explanation
Master Note for SSL Configuration in Fusion Middleware 11g [ID 1218695.1]
regards

Unexpected Swap Usage in Oracle RAC

Similar Messages

Maybe you are looking for