Rac woes
I spent a a about 16 hours setting up a rac enviornment, everything went OK, not to mention this procedure on linux is way way to complex.. but anyway I got clusterware installed and everything was fine. I accidentally rebooted rac node 2. After the reboot it was unable to mount the ocfs2 partition. anybody know why this would happen? I then decided to reboot node 1 and now both nodes are unable to mount the ocfs partition. This partition ofcs partition is on an ISCSI san. this REALLY REALLY SUCKS! It seems to be that this technology is really flaky and it should be more robust that this. sorry if I sound upset, its 1230 in the morning.
Any ideas, especially on keeping iscsi mapping correctly. it seems that that nay have been the problem. I don't even know where to begin troublesshooting this awful mess.
:(
fstab entry
[root@rh4rac1 ~]# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0
LABEL=oracrsfiles /u02/oradata/orcl ocfs2 _netdev,datavolume,nointr 0 0
/dev/hdc /media/cdrom auto pamconsole,exec,noauto,managed 0 0
/dev/fd0 /media/floppy auto pamconsole,exec,noauto,managed 0 0
OCFS IS ACTIVE
[root@rh4rac1 ~]# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold: 61
Network idle timeout: 10000
Network keepalive delay: 5000
Network reconnect delay: 2000
Checking O2CB heartbeat: Not active <--- does this say active only when both nodes are sharing a ocfs partition?
lsmod output
[root@rh4rac1 ~]# lsmod |grep ocfs
ocfs2_dlmfs 28816 1
ocfs2_dlm 253712 1 ocfs2_dlmfs
ocfs2_nodemanager 277032 5 ocfs2_dlmfs,ocfs2_dlm
configfs 32788 2 ocfs2_nodemanager
iscsi-ls -l output, all iscsi targets are still available, even the crs targe, which was specificly formatted with ocfs2
[root@rh4rac1 ~]# iscsi-ls
SFNet iSCSI Driver Version ...4:0.1.11-6(03-Aug-2007)
TARGET NAME : iqn.2006-01.com.openfiler:rac1.asm4
TARGET ALIAS :
HOST ID : 1
BUS ID : 0
TARGET ID : 0
mount -a no longer mounting ocfs2 partition
[root@rh4rac1 ~]# mount -a
mount.ocfs2: Device name specified was not found while opening device LABEL=oracrsfiles
I have not performed a up2date so the stock kernel is the same one, good thinking, that did happen to be before, however when the new kernel loaded I was not able to even start o2cb
[root@rh4rac1 ~]# uname -r
2.6.9-67.EL
below is the whole entry in /var/log/messages related to ocfs, as you can see it at around 1230am, I successfully unmounted the ocfs partition only to fail in remounting it.
Jan 25 22:00:11 rh4rac1 kernel: (8388,0):ocfs2_replay_journal:1191 Recovering node 1 from slot 0 on device (8,65)
Jan 25 22:17:42 rh4rac1 ocfs2: Stopping Oracle Cluster File System (OCFS2)
Jan 25 22:17:47 rh4rac1 kernel: ocfs2: Unmounting device (8,65) on (node 0)
Jan 25 22:17:48 rh4rac1 ocfs2: succeeded
Jan 25 22:17:48 rh4rac1 ocfs2:
Jan 25 22:17:48 rh4rac1 ocfs2:
Jan 25 22:17:48 rh4rac1 rc: Stopping ocfs2: succeeded
Jan 25 22:21:37 rh4rac1 o2cb: Loading module "ocfs2_nodemanager":
Jan 25 22:21:37 rh4rac1 o2cb: Loading module "ocfs2_dlm":
Jan 25 22:21:37 rh4rac1 o2cb: Loading module "ocfs2_dlmfs":
Jan 25 22:21:37 rh4rac1 o2cb: Mounting ocfs2_dlmfs filesystem at /dlm:
Jan 25 22:21:37 rh4rac1 o2cb: Starting O2CB cluster ocfs2:
Jan 25 22:21:39 rh4rac1 mount: mount.ocfs2
Jan 25 22:21:39 rh4rac1 ocfs2: Unable failed
Jan 25 22:35:01 rh4rac1 kernel: ocfs2_dlm: Nodes in domain ("32DAC1FBE0924FA18067D4F43A0933A9"): 0
Jan 25 22:35:01 rh4rac1 kernel: ocfs2: Mounting device (8,49) on (node 0, slot 0)
Jan 25 22:35:41 rh4rac1 kernel: ocfs2_dlm: Node 1 joins domain 32DAC1FBE0924FA18067D4F43A0933A9
Jan 25 22:35:41 rh4rac1 kernel: ocfs2_dlm: Nodes in domain ("32DAC1FBE0924FA18067D4F43A0933A9"): 0 1
Jan 26 00:11:45 rh4rac1 kernel: ocfs2_dlm: Node 1 leaves domain 32DAC1FBE0924FA18067D4F43A0933A9
Jan 26 00:11:45 rh4rac1 kernel: ocfs2_dlm: Nodes in domain ("32DAC1FBE0924FA18067D4F43A0933A9"): 0
Jan 26 00:21:12 rh4rac1 ocfs2: Stopping Oracle Cluster File System (OCFS2)
Jan 26 00:21:13 rh4rac1 ocfs2: Unable failed
Jan 26 00:21:13 rh4rac1 ocfs2:
Jan 26 00:21:13 rh4rac1 ocfs2:
Jan 26 00:21:18 rh4rac1 ocfs2: Retry stopping Oracle Cluster File System (OCFS2)
Jan 26 00:21:22 rh4rac1 kernel: ocfs2: Unmounting device (8,49) on (node 0)
Jan 26 00:21:24 rh4rac1 ocfs2: succeeded
Jan 26 00:21:24 rh4rac1 ocfs2:
Jan 26 00:21:24 rh4rac1 ocfs2:
Jan 26 00:21:24 rh4rac1 rc: Stopping ocfs2: succeeded
Jan 26 00:25:11 rh4rac1 o2cb: Loading module "ocfs2_nodemanager":
Jan 26 00:25:11 rh4rac1 o2cb: Loading module "ocfs2_dlm":
Jan 26 00:25:11 rh4rac1 o2cb: Loading module "ocfs2_dlmfs":
Jan 26 00:25:11 rh4rac1 o2cb: Mounting ocfs2_dlmfs filesystem at /dlm:
Jan 26 00:25:11 rh4rac1 o2cb: Starting O2CB cluster ocfs2:
Jan 26 00:25:15 rh4rac1 mount: mount.ocfs2
Jan 26 00:25:16 rh4rac1 ocfs2: Unable failed
the following is the crsinfo iscsi target
TARGET NAME : iqn.2006-01.com.openfiler:rac1.crsinfo
TARGET ALIAS :
HOST ID : 5
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 192.168.128.10:3260,1
SESSION STATUS : ESTABLISHED AT Sat Jan 26 00:24:08 CST 2008
SESSION ID : ISID 00023d000001 TSIH 5500
DEVICE DETAILS:
LUN ID : 0
Vendor: Openfile Model: Virtual Model: disk Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
page83 type1: 4f70656e66696c000000000006000000f42900000f000000
page80: 514548745a582d4b7731310a
Device: /dev/sde
after further review, there seems to be absolutley no configuration on it. Its suppose to have a primary partition /dev/sde1 but its emty
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): p
Disk /dev/sde: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Device Boot Start End Blocks Id System
Command (m for help): p
Disk /dev/sde: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Device Boot Start End Blocks Id System
Command (m for help):
output of fdisk -l |grep sd |sort
/dev/sdb1 1 15134 121563823+ 83 Linux
/dev/sdc1 1 15134 121563823+ 83 Linux
/dev/sdd1 1 15134 121563823+ 83 Linux
/dev/sdf1 1 15134 121563823+ 83 Linux
Disk /dev/sda: 107.3 GB, 107374182400 bytes
Disk /dev/sdb: 124.4 GB, 124486942720 bytes
Disk /dev/sdc: 124.4 GB, 124486942720 bytes
Disk /dev/sdd: 124.4 GB, 124486942720 bytes
Disk /dev/sde: 2147 MB, 2147483648 bytes
Disk /dev/sdf: 124.4 GB, 124486942720 bytes
if seems all outher drives have their primary partition except the designated ocfs drive /dev/sde in this case. Any idea how this could have gotten erased?
this is so frustrating, setting up an RAC enviornment is complex but it seems extremely fragile also.
Message was edited by:
user616818
Similar Messages
-
MULTIPLE USERS 10G RAC ORACLE_HOME INSTALL WITH ASM/CRS
Hi,
We need to install multiple 10g RAC databases on a two node Sun servers. Below is our configuration:
1) Sun Solaris (ver 10) with Sun Cluster 3.2
2) One ASM/CRS install (by 1 OS account)
3) Four ORACLE_HOME 10g database install (by 4 different OS user accounts)
We would like to use one ASM instance for all four databases with appropriate privileges.
OS User: OS Group
======== =========
oraasm dbaasm - (ASM and CRS install owner)
ora1 dbaora1 - first db owner
ora2 dbaora2 - second db owner
ora3 dbaora3 - third db owner
ora4 dbaora4 - fourth db owner
I understand that certain privileges need to be shared between ASM/CRS and DB owners. Please let me know the steps to be followed to complete this install.
Thanks in advance.Hi
Please read that: Documentation http://download.oracle.com/docs/html/B10766_08/intro.htm
- You can install and operate multiple Oracle homes and different versions of Oracle cluster database software on the same computer as described in the following points:
-You can install multiple Oracle Database 10g RAC homes on the same node. The multiple homes feature enables you to install one or more releases on the same machine in multiple Oracle home directories. However, each node can have only one CRS home.
-In addition, you cannot install Oracle Database 10g RAC into an existing single-instance Oracle home. If you have an Oracle home for Oracle Database 10g, then use a different Oracle home, and one that is available across the entire cluster for your new installation. Similarly, if you have an Oracle home for an earlier Oracle cluster database software release, then you must also use a different home for the new installation.
If the OUI detects an earlier version of a database, then the OUI asks you about your upgrade preferences. You have the option to upgrade one of the previous-version databases with DBUA or to create a new database using DBCA. The information collected during this dialog is passed to DBUA or DBCA after the software is installed.
- You can use the OUI to complete some of the de-install and re-install steps for Oracle Database 10g Real Application Clusters if needed.
Note:
Do not move Oracle binaries from one Oracle home to another because this causes dynamic link failures.
. If you are using ASM with Oracle database instances from multiple database homes on the same node, then Oracle recommends that you run the ASM instance from an Oracle home that is distinct from the database homes. In addition, the ASM home should be installed on every cluster node. This prevents the accidental removal of ASM instances that are in use by databases from other homes during the de-installation of a database's Oracle home. -
Error while running runcluvfy.sh(11g RAC on CentOS 5(RHEL 5))
Oracle Version: 11G
Operating System: Centos 5 (RHEL 5) : Linux centos51-rac-1 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:19:18 EDT 2009 i686 i686 i386 GNU/Linux
Question (including full error messages and setup scripts where applicable):
I am attempting to install oracle 11g in a RAC configuration with Centos 5 (redhat 5) as the operating system. I get the following error
ERROR : Cannot Identify the operating system. Ensure that the correct software is being executed for this operating system
Verification cannot complete
I get this error message when I run runcluvfy.sh, to verify the my configuration is clusterable. I don't know why.
I edited the /etc/redhat-release and entered echo "Red Hat Enterprise Linux AS release 4 (Nahant Update 7)" to attempt to fool the installer into thinking its red hat 4.
But still shows the same message.
Anyone knows how to fix this ?
Please help me.http://www.idevelopment.info/data/Oracle/DBA_tips/Linux/LINUX_20.shtml
runcluvfy.sh will not work on centos because the cluster verification utility checks the operating system version using the redhat-release packag and centos do this with his packages, so you must install and use redhat-release package
Get rpm-build to be able to build rpm’s:
[root@centos5 ~]# yum install rpm-build
Get source rpm of redhat-release
[root@centos5 ~]# wget ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/SRPMS/redhat-release-5Server-5.1.0.2.src.rpm
Build package:
[root@centos5 ~]# rpmbuild –rebuild redhat-release-5Server-5.1.0.2.src.rpm
Install newly generated rpm:
[root@centos5 ~]# rpm -Uvh –force /usr/src/redhat/RPMS/i386/redhat-release-5Server-5.1.0.2.i386.rpm -
Continual Mac woes (no question, just a rant)
It's Tuesday, and I am having terrible problems with my Mac. But then, why should Tuesday be different from any other day of the week.
Here is a typical day for me. The computer appears to be working OK. I need to watch a DVD for my work. I turn on DVD player, and put one in. The machine can't read the disc. It clicks and whirls, but the icon does not show up on the desktop. Meanwhile, so distressed is the machine that it freaks out. What was up until now was a fluidly operating machine suddenly reverts back to its old ways (i.e., its ways of two days ago). The hold ups and spinning pinwheels begin to eat of hours of my work day. (Remember the old days when computers made life easier?) The machine becomes sticky, gummy. Oh, I can move the curser and it seems to work for a second but then gets stuck in the dock, which explodes in icons and then freezes for five minutes. Yes. Five minutes.
Would love to use Force Quit, but the cursor is spinning, and nothing is responding. Funny about that old Mac. You can't force quit Force Quit. I guess I need to leave it open all the time.
Of course, FQ usually works on Safari. I have never just "quit" Safari. It always requires Force Quit, otherwise I can't turn off my computer. It stalls shut down.
Now I have a DVD trapped in there and can't get it out. [But I just got an answer from another posting.]
In the old macs, there used to be a pin hole you could stick a needle into ... can't find one on my flatpanel iMac.
I bought my Apple flat panel iMac in August of 2002. Yes, I know that that is a long time to have a computer, but I am not rich nor attached to a corporation that can splurge on computers. The first weekend I had the machine, I had three kernal panics.
Among the other problems I have documented are the following: the dock hiding itself unbidden and other features checking and unchecking themselves (Aug 2002); bus errors connected with OS 9 (Sept); some problems that inspired the tech person (Eric)) to talk me through deleting my user i.d., resulting in the loss of two months worth of e-mail (Thursday, 12 September); Preview problems (September); a bizarre box with an unmovable and undeletable red stop sign in it that no tech person or other Mac user I know had ever hear of (Monday 30 September); printing problems; computer won't shut down, numerous disconnection errors, which turned out to be caused by an OS X update (beginning December, 2002, or later); Kernel panics (Feb); computer won't shut down (March); Faxstexx problems, program won't allow me to set it up, finally just deleted the software (April); keys like "V" freeze and repeat endlessly (May 21); DVD Player freezes (May); Safari and Mail begin quitting unexpectedly (May); cursor begins to blink and fade out, plus odd sounds come out of the speakers, a constant error beeping (Sept 9); DVD Player problems (Oct 4).
I called AppleCare while I had it about once a week (the total between August 2002 and the time it ran out was about 155 calls). Naturally, some of these calls are motivated by user error. On the other hand, many of the issues I have called about were unprecedented as far as the Tech person was concerned, such as the blinking mouse, the red stop sign, and the DVD Player woes.
Things improved with Panther, but in Tiger many of the same old issues have returned.
I have been having so many problems with my Mac that I once wrote a letter to the company asking when do I qualify for a new replacement machine. I never received an answer, but I felt better for about a day. Then I turned on my Mac again.The spinning ball of death as we used to call it is often caused by a lack of RAM, it is hard to be sure as I am not working on your machine, but sometimes things can be improved with additional RAM, it makes it seem like a whole new computer.
A lot of your problems sound like stuff that can be fixed easily enough, and although frustrating things happen here and there with updates. It sounds like you are in fairly good spirits with it all, I would suggest just researching a bit more into maintenance you can do to help maintain the computer and educate yourself a bit more (sounds like you already have learned quite a bit along the way) and you will find a lot of these issues take you a few seconds to rid yourself of. I would start by making sure you are repairing permissions regularly and running the most up to date software. If a lot of problems persist, try creating a second user that is a "test" user to see if the problem is replicated on that user (don't delete your other one, but if you do find the problem not on the other user, you might have a corrupt user, however you don't have to lose all your emails there are plenty of ways to back it up and import it in, or even just bring the entire Mail folder from your library over to the new user). Another thing you can do if you find a lot of system problems is archive and install the OS, it takes a bit of time, but doing it overnight shouldn't be an issue, and you won't lose any of your stuff. -
Error in Creation of Dataguard for RAC
My pfile of RAC looks like:
RACDB2.__large_pool_size=4194304
RACDB1.__large_pool_size=4194304
RACDB2.__shared_pool_size=92274688
RACDB1.__shared_pool_size=92274688
RACDB2.__streams_pool_size=0
RACDB1.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/RACDB/adump'
*.background_dump_dest='/u01/app/oracle/admin/RACDB/bdump'
*.cluster_database_instances=2
*.cluster_database=true
*.compatible='10.2.0.1.0'
*.control_files='+DATA/racdb/controlfile/current.260.627905745','+FLASH/racdb/controlfile/current.256.627905753'
*.core_dump_dest='/u01/app/oracle/admin/RACDB/cdump'
*.db_block_size=8192
*.db_create_file_dest='+DATA'
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_file_name_convert='+DATA/RACDB','+DATADG/RACDG'
*.db_name='RACDB'
*.db_recovery_file_dest='+FLASH'
*.db_recovery_file_dest_size=2147483648
*.dispatchers='(PROTOCOL=TCP) (SERVICE=RACDBXDB)'
*.fal_client='RACDB'
*.fal_server='RACDG'
RACDB1.instance_number=1
RACDB2.instance_number=2
*.job_queue_processes=10
*.log_archive_config='DG_CONFIG=(RACDB,RACDG)'
*.log_archive_dest_1='LOCATION=+FLASH/RACDB/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=RACDB'
*.log_archive_dest_2='SERVICE=RACDG VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=RACDG'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='DEFER'
*.log_archive_format='%t_%s_%r.arc'
*.log_file_name_convert='+DATA/RACDB','+DATADG/RACDG'
*.open_cursors=300
*.pga_aggregate_target=16777216
*.processes=150
*.remote_listener='LISTENERS_RACDB'
*.remote_login_passwordfile='exclusive'
*.service_names='RACDB'
*.sga_target=167772160
*.standby_file_management='AUTO'
RACDB2.thread=2
RACDB1.thread=1
*.undo_management='AUTO'
RACDB2.undo_tablespace='UNDOTBS2'
RACDB1.undo_tablespace='UNDOTBS1'
*.user_dump_dest='/u01/app/oracle/admin/RACDB/udump'
My pfile of Dataguard Instance in nomount state looks like:
RACDG.__db_cache_size=58720256
RACDG.__java_pool_size=4194304
RACDG.__large_pool_size=4194304
RACDG.__shared_pool_size=96468992
RACDG.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/RACDG/adump'
*.background_dump_dest='/u01/app/oracle/admin/RACDG/bdump'
##*.cluster_database_instances=2
##*.cluster_database=true
*.compatible='10.2.0.1.0'
##*.control_files='+DATA/RACDG/controlfile/current.260.627905745','+FLASH/RACDG/controlfile/current.256.627905753'
*.core_dump_dest='/u01/app/oracle/admin/RACDG/cdump'
*.db_block_size=8192
*.db_create_file_dest='+DATADG'
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_file_name_convert='+DATADG/RACDG','+DATA/RACDB'
*.db_name='RACDB'
*.db_recovery_file_dest='+FLASHDG'
*.db_recovery_file_dest_size=2147483648
*.dispatchers='(PROTOCOL=TCP) (SERVICE=RACDGXDB)'
*.FAL_CLIENT='RACDG'
*.FAL_SERVER='RACDB'
*.job_queue_processes=10
*.LOG_ARCHIVE_CONFIG='DG_CONFIG=(RACDB,RACDG)'
*.log_archive_dest_1='LOCATION=+FLASHDG/RACDG/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=RACDG'
*.log_archive_dest_2='SERVICE=RACDB VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=RACDB'
*.LOG_ARCHIVE_DEST_STATE_1='ENABLE'
*.LOG_ARCHIVE_DEST_STATE_2='ENABLE'
*.log_archive_format='%t_%s_%r.arc'
*.log_file_name_convert='+DATADG/RACDG','+DATA/RACDB'
*.open_cursors=300
*.pga_aggregate_target=16777216
*.processes=150
##*.remote_listener='LISTENERS_RACDG'
*.remote_login_passwordfile='exclusive'
SERVICE_NAMES='RACDG'
sga_target=167772160
standby_file_management='auto'
undo_management='AUTO'
undo_tablespace='UNDOTBS1'
user_dump_dest='/u01/app/oracle/admin/RACDG/udump'
DB_UNIQUE_NAME=RACDG
and here is what I am doing on the standby location:
[oracle@dg01 ~]$ echo $ORACLE_SID
RACDG
[oracle@dg01 ~]$ rman
Recovery Manager: Release 10.2.0.1.0 - Production on Tue Jul 17 21:19:21 2007
Copyright (c) 1982, 2005, Oracle. All rights reserved.
RMAN> connect auxiliary /
connected to auxiliary database: RACDG (not mounted)
RMAN> connect target sys/xxxxxxx@RACDB
connected to target database: RACDB (DBID=625522512)
RMAN> duplicate target database for standby;
Starting Duplicate Db at 2007-07-17 22:27:08
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=156 devtype=DISK
contents of Memory Script:
restore clone standby controlfile;
sql clone 'alter database mount standby database';
executing Memory Script
Starting restore at 2007-07-17 22:27:10
using channel ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece /software/backup/ctl4.ctl
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/software/backup/ctl4.ctl tag=TAG20070717T201921
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:23
output filename=+DATADG/racdg/controlfile/current.275.628208075
output filename=+FLASHDG/racdg/controlfile/backup.268.628208079
Finished restore at 2007-07-17 22:27:34
sql statement: alter database mount standby database
released channel: ORA_AUX_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 07/17/2007 22:27:43
RMAN-05501: aborting duplication of target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/undotbs2.265.627906771 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/example.264.627905917 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/users.259.627905395 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/sysaux.257.627905385 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/undotbs1.258.627905395 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/system.256.627905375 conflicts with a file used by the target database
RMAN>
Any help to clear this error will be apprecited.......
Message was edited by:
Bal
nullHi
Thanks everybody for helping me on this issue...........
As suggested, I had taken the parameter log_file_name_convert and db_file_name_convert out of my RAC primary database but still I am getting the same error.
Any help will be appriciated..............
SQL> show parameter convert
NAME TYPE VALUE
db_file_name_convert string
log_file_name_convert string
SQL>
oracle@dg01<3>:/u01/app/oracle> rman
Recovery Manager: Release 10.2.0.1.0 - Production on Wed Jul 18 17:07:49 2007
Copyright (c) 1982, 2005, Oracle. All rights reserved.
RMAN> connect auxiliary /
connected to auxiliary database: RACDB (not mounted)
RMAN> connect target sys/xxx@RACDB
connected to target database: RACDB (DBID=625522512)
RMAN> duplicate target database for standby;
Starting Duplicate Db at 2007-07-18 17:10:53
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=156 devtype=DISK
contents of Memory Script:
restore clone standby controlfile;
sql clone 'alter database mount standby database';
executing Memory Script
Starting restore at 2007-07-18 17:10:54
using channel ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece /software/backup/ctl5.ctr
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/software/backup/ctl5.ctr tag=TAG20070718T170529
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:33
output filename=+DATADG/racdg/controlfile/current.275.628208075
output filename=+FLASHDG/racdg/controlfile/backup.268.628208079
Finished restore at 2007-07-18 17:11:31
sql statement: alter database mount standby database
released channel: ORA_AUX_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 07/18/2007 17:11:43
RMAN-05501: aborting duplication of target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/undotbs2.265.627906771 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/example.264.627905917 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/users.259.627905395 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/sysaux.257.627905385 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/undotbs1.258.627905395 conflicts with a file used by the target database
RMAN-05001: auxiliary filename +DATA/racdb/datafile/system.256.627905375 conflicts with a file used by the target database -
How to install 11gR2 RAC on 64 bit linux OS
I am completely new to this topic of RAC and need to be installing and standing up RAC on Linux 64 bit OS . I have good knowledge of installing oracle database ENTERPRISE version 11gR2.
Can you guide me as to how to start. I am looking for leads. Probably we will have 2 nodes.
Thank you very much for helping me in advanceIf you are a My Oracle Support (Metalink) user, go check out these two notes created by the Oracle RAC Assurance Team. They are excellent.
NOTE: 810394.1 RAC Assurance Support Team: RAC Starter Kit and Best Practices (Generic)
NOTE: 811306.1 RAC Assurance Support Team: RAC Starter Kit (Linux)
In the Linux note mentioned above there is a link to a Linux Step by Step Instruction Guide. This step by step instruction guide is the best start to finish document I've seen for how to set-up and install Oracle RAC. I believe the guide is written for installing release 11.2.0.2. -
In oracle rac, If user query a select query and in processing data is fetched but in the duration of fetching the particular node is evicted then how failover to another node internally?
The query is re-issued as a flashback query and the client process can continue to fetch from the cursor. This is described in the Net Services Administrators Guide, the section on Transparent Application Failover.
-
RMAN, RAC, NFS, and server lock ups
Good day. My environment is:
--a 2-node RAC
--Enterprise Edition 11.2.0.3
--RHEL 5.1
The goal is to use RMAN to push backups to a shared NFS mount (on a different server). Both nodes will have access to this location (in the event one node goes down, the other can still run backups). Easy, right?
Wrong.
I've tried every NFS mount option in the book. Most work just fine, some don't. When I use the recommended NFS mount options:
rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp, vers=3,timeo=600, actimeo=0
or
rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,forcedirectio, vers=3,suid
The mount works normally. I can "ls" and "mkdir" and "touch" and "vi" and "cp" files back and forth from the NFS backup location to the RAC node all day long. No problems. However, when I try to do almost anything in RMAN which requires writing to the NFS backup location such as the command "backup archive all delete input;" (or even things as simple as a Crosscheck or RMAN configuration change which writes any changes back to the autobackup ControlFile) the node locks up. There are no errors (or if there are, I don't know where to find them), even when I use RMAN log.
Just to recap: I run a Crosscheck (or any RMAN process that writes to the NFS backup location), the node will lock up, and I can let it sit for a day, inaccessible, with CRSCTL on the other node saying it's offline, and the node will never come out of a "frozen" state. It cannot be pinged or connected to.
I think I can safely rule out NFS mount options at this point.
I understand (after extensive reading of MOS docs and testing) that RAC RMAN can and does suffer from inefficient I/O when writing to an NFS mount. I don't think that's the culprit either. The autobackup ControlFile is not that big and I cannot see how running a simple Crosscheck would lock an entire node.
I am hoping someone has encountered this in the past and hopefully it's just a simple misconfiguration somewhere.My NFS line in /etc/fstab is (these options are for supporting 11.2.0.3, 11.1.0.7, and 10.2.0.4/5 simultaneously): server.domain:/NFS_Export /backup nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0
Before you installed GI, did you by chance do a yum update? I've encountered a similar issue which ended up being due to mkinitrd creating a corrupted kernel; mkinitrd is invoked during the GI installation when the ADVM drivers are added and in my case mkinitrd created a new kernel prior to the new kernel being installed. Second to that, make sure you have the matching kernel headers to your kernel version. If they are different then you could probably get away with just creating a new kernel with mkinitrd and relinking GI/RDBMS homes, but be prepared to wipe GI and reinstall. -
Gns is getting failed with error CRS-2632 during RAC installation
hello guys i am new to oracle RAC and i am trying to configure two node ORACLE 11G R2 RAC setup on OEL 5.4 using GNS Every things works great till I execute
root.sh script on the first node
It gives me error
CRS-2674: Start of 'ora.gns' on 'host01' failed
CRS-2632: There are no more servers to try to place resource 'ora.gns' on that would satisfy its placement policy
start gns ... failed
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... failed
Updating inventory properties for clusterware
Starting Oracle Universal Installer...
When i check status of cluster recourse i get this as output
[root@host01 ~]# crs_stat -t
Name Type Target State Host
ora.DATA.dg ora....up.type ONLINE ONLINE host01
ora....N1.lsnr ora....er.type OFFLINE OFFLINE
ora....N2.lsnr ora....er.type OFFLINE OFFLINE
ora....N3.lsnr ora....er.type OFFLINE OFFLINE
ora.asm ora.asm.type ONLINE ONLINE host01
ora.eons ora.eons.type ONLINE ONLINE host01
ora.gns ora.gns.type ONLINE OFFLINE
ora.gns.vip ora....ip.type ONLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE host01
ora.host01.gsd application OFFLINE OFFLINE
ora.host01.ons application ONLINE ONLINE host01
ora.host01.vip ora....t1.type ONLINE ONLINE host01
ora....network ora....rk.type ONLINE ONLINE host01
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE host01
ora....ry.acfs ora....fs.type OFFLINE OFFLINE
ora.scan1.vip ora....ip.type OFFLINE OFFLINE
ora.scan2.vip ora....ip.type OFFLINE OFFLINE
ora.scan3.vip ora....ip.type OFFLINE OFFLINE
These are my GNS configuration file entries
vi /var/named/chroot/etc/named.conf
options {
listen-on port 53 { 192.9.201.59; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
allow-query { any; };
allow-query-cache { any; };
zone "." IN {
type hint;
file "named.ca";
zone "localdomain" IN {
type master;
file "localdomain.zone";
allow-update { none; };
zone "localhost" IN {
type master;
file "localhost.zone";
allow-update { none; };
zone "0.0.127.in-addr.arpa" IN {
type master;
file "named.local";
allow-update { none; };
zone "0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
type master;
file "named.ip6.local";
allow-update { none; };
zone "255.in-addr.arpa" IN {
type master;
file "named.broadcast";
allow-update { none; };
zone "0.in-addr.arpa" IN {
type master;
file "named.zero";
allow-update { none; };
zone "example.com" IN {
type master;
file "forward.zone";
allow-transfer { 192.9.201.180; };
zone "201.9.192.in-addr.arpa" IN {
type master;
file "reverse.zone";
zone "0.0.10.in-addr.arpa" IN {
type master;
file "reverse1.zone";
vi /var/named/chroot/var/named/forward.zone
$TTL 86400
@ IN SOA server1.example.com. root.server1.example.com. (
42 ; serial (d. adams)
3H ; refresh
15M ; retry
1W ; expiry
1D ) ; minimum
IN NS server1
IN A 192.9.201.59
server1 IN A 192.9.201.59
host01 IN A 192.9.201.181
host02 IN A 192.9.201.182
host03 IN A 192.9.201.183
openfiler IN A 192.9.201.184
host01-priv IN A 10.0.0.2
host02-priv IN A 10.0.0.3
host03-priv IN A 10.0.0.4
vi /var/named/chroot/var/named/reverse.zone
$ORIGIN cluster01.example.com.
@ IN NS cluster01-gns.cluster01.example.com.
cluster01-gns IN A 192.9.201.180
$TTL 86400
@ IN SOA server1.example.com. root.server1.example.com. (
1997022700 ; Serial
28800 ; Refresh
14400 ; Retry
3600000 ; Expire
86400 ) ; Minimum
IN NS server1.example.com.
59 IN PTR server1.example.com.
184 IN PTR openfiler.example.com.
181 IN PTR host01.example.com.
182 IN PTR host02.example.com.
183 IN PTR host03.example.com.
vi /var/named/chroot/var/named/reverse1.zone
$TTL 86400
@ IN SOA server1.example.com. root.server1.example.com. (
1997022700 ; Serial
28800 ; Refresh
14400 ; Retry
3600000 ; Expire
86400 ) ; Minimum
IN NS server1.example.com.
2 IN PTR host01-priv.example.com.
3 IN PTR host02-priv.example.com.
4 IN PTR host03-priv.example.com.
Please suggest me what i am doing wrong
Edited by: 1001408 on Apr 21, 2013 9:17 AM
Edited by: 1001408 on Apr 21, 2013 9:22 AMHello guys finally i find mistake i was doing
while configuring Public Ip for the nodes i was not giving Default Gateway .I was assuming as all these machine is in same network with same Ip range so they would not be needing Gateway but here my assumption mismatch with oracle well finally happy to see 11G r2 with GNS running on my personal laptop.
cheers
Rahul -
Oracel 11gR1 RAC Cluster issue
We have 2-node Oracle 11gR2 RAC on HP-UX 11.31 environment. It was running lase 2 month without any issue.
We got some netconfig issue, and node-1 got rebooted today. after the reboot cluster didn't not start on node-1, database is running on node-2.
grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager
grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=301, loc=kgfokge
AMDU-00301: Unable to open file tmp-AMIPOCR01.ocr
AMDU-00204: Disk N0002 is in currently mounted diskgroup AMIPOCR01
AMDU-00201: Disk N0002: '/dev/rdisk/ora_OCR
] [8]
grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ olsnodes -n
hublhp4 1
hublhp5 2
any idea please.
Edited by: ManoRangasamy on Jul 5, 2011 6:38 PMHi,
Please post the alertlog ASM from node 1, crsd.log and ocssd.log from node 1
It might be because node 1 can't see asm disk or permission accidentally changed when the node rebooted
Cheers -
Failed root.sh on 1st Node of 11.2.0.2.0 RAC on HP-UX 11.31 Itanium 64
Started with 11.2.0.2.0 Grid Installation for 2 Node RAC on HP-UX 11.31 Itanium 64.
Copying Software to remote node & linking libraries were successfully without any issue (upto 76%). But got issue while executing root.sh on Node1
sph1erp:/oracle/11.2.0/grid #sh root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'sph1erp'
CRS-2676: Start of 'ora.mdnsd' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'sph1erp'
CRS-2676: Start of 'ora.gpnpd' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'sph1erp'
CRS-2672: Attempting to start 'ora.gipcd' on 'sph1erp'
CRS-2676: Start of 'ora.gipcd' on 'sph1erp' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'sph1erp'
CRS-2672: Attempting to start 'ora.diskmon' on 'sph1erp'
CRS-2676: Start of 'ora.diskmon' on 'sph1erp' succeeded
CRS-2676: Start of 'ora.cssd' on 'sph1erp' succeeded
ASM created and started successfully.
Disk Group OCRVOTE created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk ab847ed2b4f04f2dbfb875226d2bb194.
Successful addition of voting disk 85c05a5b30384f8dbff48cc069de7a7c.
Successful addition of voting disk 649196fbdd614f9cbf26a9a0e6670a6e.
Successful addition of voting disk 8815dfcee2e64f64bf00b9c76626ab41.
Successful addition of voting disk 8ce55fe5534f4f77bfa9f54187592707.
Successfully replaced voting disk group with +OCRVOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE ab847ed2b4f04f2dbfb875226d2bb194 (/dev/oracle/ocrvote1) [OCRVOTE]
2. ONLINE 85c05a5b30384f8dbff48cc069de7a7c (/dev/oracle/ocrvote2) [OCRVOTE]
3. ONLINE 649196fbdd614f9cbf26a9a0e6670a6e (/dev/oracle/ocrvote3) [OCRVOTE]
4. ONLINE 8815dfcee2e64f64bf00b9c76626ab41 (/dev/oracle/ocrvote4) [OCRVOTE]
5. ONLINE 8ce55fe5534f4f77bfa9f54187592707 (/dev/oracle/ocrvote5) [OCRVOTE]
Located 5 voting disk(s).
Start of resource "ora.cluster_interconnect.haip" failed
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'sph1erp'
CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
Start action for HAIP aborted
CRS-2674: Start of 'ora.cluster_interconnect.haip' on 'sph1erp' failed
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'sph1erp'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'sph1erp' succeeded
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Clusterware stack
Failed to start High Availability IP at /oracle/11.2.0/grid/crs/install/crsconfig_lib.pm line 1046.
*/oracle/11.2.0/grid/perl/bin/perl -I/oracle/11.2.0/grid/perl/lib -I/oracle/11.2.0/grid/crs/install /oracle/11.2.0/grid/crs/install/rootcrs.pl execution failed*
sph1erp:/oracle/11.2.0/grid #
Last few lines from CRS Log for node 1, where error came
[ctssd(6467)]CRS-2401:The Cluster Time Synchronization Service started on host sph1erp.
2011-02-25 23:04:16.491
[oracle/11.2.0/grid/bin/orarootagent.bin(6423)]CRS-5818:Aborted command 'start for resource: ora.cluster_interconnect.haip 1 1' for resource 'ora.cluster_int
erconnect.haip'. Details at (:CRSAGF00113:) {0:0:178} in */oracle/11.2.0/grid/log/sph1erp/agent/ohasd/orarootagent_root/orarootagent_root.log.*
2011-02-25 23:04:20.521
[ohasd(5513)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:178} in
*/oracle/11.2.0/grid/log/sph1erp/ohasd/ohasd.log.*
Few lines from */oracle/11.2.0/grid/log/sph1erp/agent/ohasd/orarootagent_root/orarootagent_root.log.*
=====================================================================================================
2011-02-25 23:04:16.823: [ USRTHRD][16] {0:0:178} Starting Probe for ip 169.254.74.54
2011-02-25 23:04:16.823: [ USRTHRD][16] {0:0:178} Transitioning to Probe State
2011-02-25 23:04:17.177: [ USRTHRD][15] {0:0:178} [NetHAMain] thread stopping
2011-02-25 23:04:17.177: [ USRTHRD][15] {0:0:178} Thread:[NetHAMain]isRunning is reset to false here
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} Thread:[NetHAMain]stop }
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} thread cleaning up
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} pausing thread
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} posting thread
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop {
2011-02-25 23:04:17.645: [ USRTHRD][16] {0:0:178} [NetHAWork] thread stopping
2011-02-25 23:04:17.645: [ USRTHRD][16] {0:0:178} Thread:[NetHAWork]isRunning is reset to false here
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop }
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop {
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop }
2011-02-25 23:04:17.891: [ora.cluster_interconnect.haip][12] {0:0:178} [start] Start of HAIP aborted
2011-02-25 23:04:17.892: [ AGENT][12] {0:0:178} UserErrorException: Locale is
2011-02-25 23:04:17.893: [ora.cluster_interconnect.haip][12] {0:0:178} [start] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
Start action for HAIP aborted
2011-02-25 23:04:17.893: [ AGFW][12] {0:0:178} sending status msg [CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the foll
owing error:
Start action for HAIP aborted
] for start for resource: ora.cluster_interconnect.haip 1 1
2011-02-25 23:04:17.893: [ora.cluster_interconnect.haip][12] {0:0:178} [start] clsn_agent::start }
2011-02-25 23:04:17.894: [ AGFW][10] {0:0:178} Agent sending reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:661
2011-02-25 23:04:18.552: [ora.diskmon][12] {0:0:154} [check] DiskmonAgent::check {
2011-02-25 23:04:18.552: [ora.diskmon][12] {0:0:154} [check] DiskmonAgent::check } - 0
2011-02-25 23:04:19.573: [ AGFW][10] {0:0:154} Agent received the message: AGENT_HB[Engine] ID 12293:669
2011-02-25 23:04:20.510: [ora.cluster_interconnect.haip][18] {0:0:178} [start] got lock
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] tryActionLock }
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] abort }
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] clsn_agent::abort }
2011-02-25 23:04:20.511: [ AGFW][18] {0:0:178} Command: start for resource: ora.cluster_interconnect.haip 1 1 completed with status: TIMEDOUT
2011-02-25 23:04:20.512: [ora.cluster_interconnect.haip][8] {0:0:178} [check] NetworkAgent::init enter {
2011-02-25 23:04:20.513: [ora.cluster_interconnect.haip][8] {0:0:178} [check] NetworkAgent::init exit }
2011-02-25 23:04:20.517: [ AGFW][10] {0:0:178} Agent sending reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:661
2011-02-25 23:04:20.519: [ USRTHRD][8] {0:0:178} Ocr Context init default level 23886304
2011-02-25 23:04:20.519: [ default][8]clsvactversion:4: Retrieving Active Version from local storage.
[ CLWAL][8]clsw_Initialize: OLR initlevel [70000]
Few lines from */oracle/11.2.0/grid/log/sph1erp/ohasd/ohasd.log.*
=====================================================================================================
2011-02-25 23:04:21.627: [UiServer][30] {0:0:180} Done for ctx=6000000002604ce0
2011-02-25 23:04:21.642: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:26.139: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:26.139: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:26.973: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:26.973: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:26.992: [UiServer][30] {0:0:181} processMessage called
2011-02-25 23:04:26.993: [UiServer][30] {0:0:181} Sending message to PE. ctx= 6000000001b440f0
2011-02-25 23:04:26.993: [UiServer][30] {0:0:181} Sending command to PE: 67
2011-02-25 23:04:26.994: [ CRSPE][29] {0:0:181} Processing PE command id=173. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:26.997: [UiServer][30] {0:0:181} Done for ctx=6000000001b440f0
2011-02-25 23:04:27.012: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:31.135: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:31.135: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:32.318: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:32.318: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:32.332: [UiServer][30] {0:0:182} processMessage called
2011-02-25 23:04:32.333: [UiServer][30] {0:0:182} Sending message to PE. ctx= 6000000001b45ef0
2011-02-25 23:04:32.333: [UiServer][30] {0:0:182} Sending command to PE: 68
2011-02-25 23:04:32.334: [ CRSPE][29] {0:0:182} Processing PE command id=174. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:32.338: [UiServer][30] {0:0:182} Done for ctx=6000000001b45ef0
2011-02-25 23:04:32.352: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:36.155: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:36.155: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:37.683: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:37.683: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:37.702: [UiServer][30] {0:0:183} processMessage called
2011-02-25 23:04:37.703: [UiServer][30] {0:0:183} Sending message to PE. ctx= 6000000002604ce0
2011-02-25 23:04:37.703: [UiServer][30] {0:0:183} Sending command to PE: 69
2011-02-25 23:04:37.704: [ CRSPE][29] {0:0:183} Processing PE command id=175. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:37.708: [UiServer][30] {0:0:183} Done for ctx=6000000002604ce0
2011-02-25 23:04:37.722: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:41.156: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:41.156: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
What could be the issue ????
Experts Please help me. Doing setup for the PRoduction Env...
Do response ASAP...... Thanks
Regards,
ManishThanks Sebastian for your input.
yes. my lan2 is used for Cluster_interconnect which is having subnet 255.255.255.240.
Below are IPs used for RAC
Public
Node1: 10.10.1.173/255.255.240.0
Node2: 10.10.1.174/255.255.240.0
Private
Node1: 10.10.16.50/255.255.255.240
Node2: 10.10.16.51/255.255.255.240
Virtual
Node1: 10.10.1.191/255.255.240.0
Node2: 10.10.1.192/255.255.240.0
SCAN (Defined in DNS)
10.10.1.193/255.255.240.0
10.10.1.194/255.255.240.0
10.10.1.195/255.255.240.0
As you said, I will scrap GI Software again & will try with 255.255.255.0.
I Believe this Redundant Interconnect and ora.cluster_interconnect.haip present in 11.2.0.2.0 Version.
Oracle says:
Redundant Interconnect without any 3rd-party IP failover technology (bond, IPMP or similar) is supported natively by Grid Infrastructure starting from 11.2.0.2. Multiple private network adapters can be defined either during the installation phase or afterward using the oifcfg. Oracle Database, CSS, OCR, CRS, CTSS, and EVM components in 11.2.0.2 employ it automatically.
Grid Infrastructure can activate a maximum of four private network adapters at a time even if more are defined. The ora.cluster_interconnect.haip resource will start one to four link local HAIP on private network adapters for interconnect communication for Oracle RAC, Oracle ASM, and Oracle ACFS etc.
Grid automatically picks link local addresses from reserved 169.254.*.* subnet for HAIP, and it will not attempt to use any 169.254.*.* address if it's already in use for another purpose. With HAIP, by default, interconnect traffic will be load balanced across all active interconnect interfaces, and corresponding HAIP address will be failed over transparently to other adapters if one fails or becomes non-communicative. .
The number of HAIP addresses is decided by how many private network adapters are active when Grid comes up on the first node in the cluster . If there's only one active private network, Grid will create one; if two, Grid will create two; and if more than two, Grid will create four HAIPs. The number of HAIPs won't change even if more private network adapters are activated later, a restart of clusterware on all nodes is required for new adapters to become effective.
In my Setup, I am having Teaming for NIC's for Public & Private Interface. So I am thinking to break teaming of NICs because HAIP internally searching for next available NIC & not getting as all 4 are already in used with OS level NIC teaming.
My only Concern is, as I am going to change subnet for the Private IPs, should I change Private IP address ????
Thanks for the Support...
Regards,
Manish -
Active session Spike on Oracle RAC 11G R2 on HP UX
Dear Experts,
We need urgent help please, as we are facing very low performance in production database.
We are having oracle 11G RAC on HP Unix environment. Following is the ADDM report. Kindly check and please help me to figure it out the issue and resolve it at earliest.
---------Instance 1---------------
ADDM Report for Task 'TASK_36650'
Analysis Period
AWR snapshot range from 11634 to 11636.
Time period starts at 21-JUL-13 07.00.03 PM
Time period ends at 21-JUL-13 09.00.49 PM
Analysis Target
Database 'MCMSDRAC' with DB ID 2894940361.
Database version 11.2.0.1.0.
ADDM performed an analysis of instance mcmsdrac1, numbered 1 and hosted at
mcmsdbl1.
Activity During the Analysis Period
Total database time was 38466 seconds.
The average number of active sessions was 5.31.
Summary of Findings
Description Active Sessions Recommendations
Percent of Activity
1 CPU Usage 1.44 | 27.08 1
2 Interconnect Latency .07 | 1.33 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Findings and Recommendations
Finding 1: CPU Usage
Impact is 1.44 active sessions, 27.08% of total activity.
Host CPU was a bottleneck and the instance was consuming 99% of the host CPU.
All wait times will be inflated by wait for CPU.
Host CPU consumption was 99%.
Recommendation 1: Host Configuration
Estimated benefit is 1.44 active sessions, 27.08% of total activity.
Action
Consider adding more CPUs to the host or adding instances serving the
database on other hosts.
Action
Session CPU consumption was throttled by the Oracle Resource Manager.
Consider revising the resource plan that was active during the analysis
period.
Finding 2: Interconnect Latency
Impact is .07 active sessions, 1.33% of total activity.
Higher than expected latency of the cluster interconnect was responsible for
significant database time on this instance.
The instance was consuming 110 kilo bits per second of interconnect bandwidth.
20% of this interconnect bandwidth was used for global cache messaging, 21%
for parallel query messaging and 7% for database lock management.
The average latency for 8K interconnect messages was 42153 microseconds.
The instance is using the private interconnect device "lan2" with IP address
172.16.200.71 and source "Oracle Cluster Repository".
The device "lan2" was used for 100% of interconnect traffic and experienced 0
send or receive errors during the analysis period.
Recommendation 1: Host Configuration
Estimated benefit is .07 active sessions, 1.33% of total activity.
Action
Investigate cause of high network interconnect latency between database
instances. Oracle's recommended solution is to use a high speed
dedicated network.
Action
Check the configuration of the cluster interconnect. Check OS setup like
adapter setting, firmware and driver release. Check that the OS's socket
receive buffers are large enough to store an entire multiblock read. The
value of parameter "db_file_multiblock_read_count" may be decreased as a
workaround.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Additional Information
Miscellaneous Information
Wait class "Application" was not consuming significant database time.
Wait class "Cluster" was not consuming significant database time.
Wait class "Commit" was not consuming significant database time.
Wait class "Concurrency" was not consuming significant database time.
Wait class "Configuration" was not consuming significant database time.
Wait class "Network" was not consuming significant database time.
Wait class "User I/O" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
Hard parsing of SQL statements was not consuming significant database time.
The database's maintenance windows were active during 100% of the analysis
period.
----------------Instance 2 --------------------
ADDM Report for Task 'TASK_36652'
Analysis Period
AWR snapshot range from 11634 to 11636.
Time period starts at 21-JUL-13 07.00.03 PM
Time period ends at 21-JUL-13 09.00.49 PM
Analysis Target
Database 'MCMSDRAC' with DB ID 2894940361.
Database version 11.2.0.1.0.
ADDM performed an analysis of instance mcmsdrac2, numbered 2 and hosted at
mcmsdbl2.
Activity During the Analysis Period
Total database time was 2898 seconds.
The average number of active sessions was .4.
Summary of Findings
Description Active Sessions Recommendations
Percent of Activity
1 Top SQL Statements .11 | 27.65 5
2 Interconnect Latency .1 | 24.15 1
3 Shared Pool Latches .09 | 22.42 1
4 PL/SQL Execution .06 | 14.39 2
5 Unusual "Other" Wait Event .03 | 8.73 4
6 Unusual "Other" Wait Event .03 | 6.42 3
7 Unusual "Other" Wait Event .03 | 6.29 6
8 Hard Parse .02 | 5.5 0
9 Soft Parse .02 | 3.86 2
10 Unusual "Other" Wait Event .01 | 3.75 4
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Findings and Recommendations
Finding 1: Top SQL Statements
Impact is .11 active sessions, 27.65% of total activity.
SQL statements consuming significant database time were found. These
statements offer a good opportunity for performance improvement.
Recommendation 1: SQL Tuning
Estimated benefit is .05 active sessions, 12.88% of total activity.
Action
Investigate the PL/SQL statement with SQL_ID "d1s02myktu19h" for
possible performance improvements. You can supplement the information
given here with an ASH report for this SQL_ID.
Related Object
SQL statement with SQL_ID d1s02myktu19h.
begin dbms_utility.validate(:1,:2,:3,:4); end;
Rationale
The SQL Tuning Advisor cannot operate on PL/SQL statements.
Rationale
Database time for this SQL was divided as follows: 13% for SQL
execution, 2% for parsing, 85% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "d1s02myktu19h" was executed 48 times and had
an average elapsed time of 7 seconds.
Rationale
Waiting for event "library cache pin" in wait class "Concurrency"
accounted for 70% of the database time spent in processing the SQL
statement with SQL_ID "d1s02myktu19h".
Rationale
Top level calls to execute the PL/SQL statement with SQL_ID
"63wt8yna5umd6" are responsible for 100% of the database time spent on
the PL/SQL statement with SQL_ID "d1s02myktu19h".
Related Object
SQL statement with SQL_ID 63wt8yna5umd6.
begin DBMS_UTILITY.COMPILE_SCHEMA( 'TPAUSER', FALSE ); end;
Recommendation 2: SQL Tuning
Estimated benefit is .02 active sessions, 4.55% of total activity.
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"fk3bh3t41101x".
Related Object
SQL statement with SQL_ID fk3bh3t41101x.
SELECT MEM.MEMBER_CODE ,MEM.E_NAME,Pol.Policy_no
,pol.date_from,pol.date_to,POL.E_NAME,MEM.SEX,(SYSDATE-MEM.BIRTH_DATE
) AGE,POL.SCHEME_NO FROM TPAUSER.MEMBERS MEM,TPAUSER.POLICY POL WHERE
POL.QUOTATION_NO=MEM.QUOTATION_NO AND POL.BRANCH_CODE=MEM.BRANCH_CODE
and endt_no=(select max(endt_no) from tpauser.members mm where
mm.member_code=mem.member_code AND mm.QUOTATION_NO=MEM.QUOTATION_NO)
and member_code like '%' || nvl(:1,null) ||'%' ORDER BY MEMBER_CODE
Rationale
The SQL spent 92% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "fk3bh3t41101x" was executed 14 times and had
an average elapsed time of 4.9 seconds.
Rationale
At least one execution of the statement ran in parallel.
Recommendation 3: SQL Tuning
Estimated benefit is .02 active sessions, 3.79% of total activity.
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"7mhjbjg9ntqf5".
Related Object
SQL statement with SQL_ID 7mhjbjg9ntqf5.
SELECT SUM(CNT) FROM (SELECT COUNT(PROC_CODE) CNT FROM
TPAUSER.TORBINY_PROCEDURE WHERE BRANCH_CODE = :B6 AND QUOTATION_NO =
:B5 AND CLASS_NO = :B4 AND OPTION_NO = :B3 AND PR_EFFECTIVE_DATE<=
:B2 AND PROC_CODE = :B1 UNION SELECT COUNT(MED_CODE) CNT FROM
TPAUSER.TORBINY_MEDICINE WHERE BRANCH_CODE = :B6 AND QUOTATION_NO =
:B5 AND CLASS_NO = :B4 AND OPTION_NO = :B3 AND M_EFFECTIVE_DATE<= :B2
AND MED_CODE = :B1 UNION SELECT COUNT(LAB_CODE) CNT FROM
TPAUSER.TORBINY_LAB WHERE BRANCH_CODE = :B6 AND QUOTATION_NO = :B5
AND CLASS_NO = :B4 AND OPTION_NO = :B3 AND L_EFFECTIVE_DATE<= :B2 AND
LAB_CODE = :B1 )
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 0% for SQL execution,
0% for parsing, 100% for PL/SQL execution and 0% for Java execution.
Rationale
SQL statement with SQL_ID "7mhjbjg9ntqf5" was executed 31 times and had
an average elapsed time of 3.4 seconds.
Rationale
Top level calls to execute the SELECT statement with SQL_ID
"a11nzdnd91gsg" are responsible for 100% of the database time spent on
the SELECT statement with SQL_ID "7mhjbjg9ntqf5".
Related Object
SQL statement with SQL_ID a11nzdnd91gsg.
SELECT POLICY_NO,SCHEME_NO FROM TPAUSER.POLICY WHERE QUOTATION_NO
=:B1
Recommendation 4: SQL Tuning
Estimated benefit is .01 active sessions, 3.03% of total activity.
Action
Investigate the SELECT statement with SQL_ID "4uqs4jt7aca5s" for
possible performance improvements. You can supplement the information
given here with an ASH report for this SQL_ID.
Related Object
SQL statement with SQL_ID 4uqs4jt7aca5s.
SELECT DISTINCT USER_ID FROM GV$SESSION, USERS WHERE UPPER (USERNAME)
= UPPER (USER_ID) AND USERS.APPROVAL_CLAIM='VC' AND USER_ID=:B1
Rationale
The SQL spent only 0% of its database time on CPU, I/O and Cluster
waits. Therefore, the SQL Tuning Advisor is not applicable in this case.
Look at performance data for the SQL to find potential improvements.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "4uqs4jt7aca5s" was executed 261 times and had
an average elapsed time of 0.35 seconds.
Rationale
At least one execution of the statement ran in parallel.
Rationale
Top level calls to execute the PL/SQL statement with SQL_ID
"91vt043t78460" are responsible for 100% of the database time spent on
the SELECT statement with SQL_ID "4uqs4jt7aca5s".
Related Object
SQL statement with SQL_ID 91vt043t78460.
begin TPAUSER.RECEIVE_NEW_FAX_APRROVAL(:V00001,:V00002,:V00003,:V0000
4); end;
Recommendation 5: SQL Tuning
Estimated benefit is .01 active sessions, 3.03% of total activity.
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"7kt28fkc0yn5f".
Related Object
SQL statement with SQL_ID 7kt28fkc0yn5f.
SELECT COUNT(*) FROM TPAUSER.APPROVAL_MASTER WHERE APPROVAL_STATUS IS
NULL AND (UPPER(CODED) = UPPER(:B1 ) OR UPPER(PROCESSED_BY) =
UPPER(:B1 ))
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "7kt28fkc0yn5f" was executed 1034 times and
had an average elapsed time of 0.063 seconds.
Rationale
Top level calls to execute the PL/SQL statement with SQL_ID
"91vt043t78460" are responsible for 100% of the database time spent on
the SELECT statement with SQL_ID "7kt28fkc0yn5f".
Related Object
SQL statement with SQL_ID 91vt043t78460.
begin TPAUSER.RECEIVE_NEW_FAX_APRROVAL(:V00001,:V00002,:V00003,:V0000
4); end;
Finding 2: Interconnect Latency
Impact is .1 active sessions, 24.15% of total activity.
Higher than expected latency of the cluster interconnect was responsible for
significant database time on this instance.
The instance was consuming 128 kilo bits per second of interconnect bandwidth.
17% of this interconnect bandwidth was used for global cache messaging, 6% for
parallel query messaging and 8% for database lock management.
The average latency for 8K interconnect messages was 41863 microseconds.
The instance is using the private interconnect device "lan2" with IP address
172.16.200.72 and source "Oracle Cluster Repository".
The device "lan2" was used for 100% of interconnect traffic and experienced 0
send or receive errors during the analysis period.
Recommendation 1: Host Configuration
Estimated benefit is .1 active sessions, 24.15% of total activity.
Action
Investigate cause of high network interconnect latency between database
instances. Oracle's recommended solution is to use a high speed
dedicated network.
Action
Check the configuration of the cluster interconnect. Check OS setup like
adapter setting, firmware and driver release. Check that the OS's socket
receive buffers are large enough to store an entire multiblock read. The
value of parameter "db_file_multiblock_read_count" may be decreased as a
workaround.
Symptoms That Led to the Finding:
Inter-instance messaging was consuming significant database time on this
instance.
Impact is .06 active sessions, 14.23% of total activity.
Wait class "Cluster" was consuming significant database time.
Impact is .06 active sessions, 14.23% of total activity.
Finding 3: Shared Pool Latches
Impact is .09 active sessions, 22.42% of total activity.
Contention for latches related to the shared pool was consuming significant
database time.
Waits for "library cache lock" amounted to 5% of database time.
Waits for "library cache pin" amounted to 17% of database time.
Recommendation 1: Application Analysis
Estimated benefit is .09 active sessions, 22.42% of total activity.
Action
Investigate the cause for latch contention using the given blocking
sessions or modules.
Rationale
The session with ID 17 and serial number 15595 in instance number 1 was
the blocking session responsible for 34% of this recommendation's
benefit.
Symptoms That Led to the Finding:
Wait class "Concurrency" was consuming significant database time.
Impact is .1 active sessions, 24.96% of total activity.
Finding 4: PL/SQL Execution
Impact is .06 active sessions, 14.39% of total activity.
PL/SQL execution consumed significant database time.
Recommendation 1: SQL Tuning
Estimated benefit is .05 active sessions, 12.5% of total activity.
Action
Tune the entry point PL/SQL "SYS.DBMS_UTILITY.COMPILE_SCHEMA" of type
"PACKAGE" and ID 6019. Refer to the PL/SQL documentation for addition
information.
Rationale
318 seconds spent in executing PL/SQL "SYS.DBMS_UTILITY.VALIDATE#2" of
type "PACKAGE" and ID 6019.
Recommendation 2: SQL Tuning
Estimated benefit is .01 active sessions, 1.89% of total activity.
Action
Tune the entry point PL/SQL
"SYSMAN.EMD_MAINTENANCE.EXECUTE_EM_DBMS_JOB_PROCS" of type "PACKAGE" and
ID 68654. Refer to the PL/SQL documentation for addition information.
Finding 5: Unusual "Other" Wait Event
Impact is .03 active sessions, 8.73% of total activity.
Wait event "DFS lock handle" in wait class "Other" was consuming significant
database time.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 8.73% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits. Refer to
Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .03 active sessions, 8.27% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits in Service
"mcmsdrac".
Recommendation 3: Application Analysis
Estimated benefit is .02 active sessions, 5.05% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits in Module "TOAD
9.7.2.5".
Recommendation 4: Application Analysis
Estimated benefit is .01 active sessions, 3.21% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits in Module
"toad.exe".
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
Finding 6: Unusual "Other" Wait Event
Impact is .03 active sessions, 6.42% of total activity.
Wait event "reliable message" in wait class "Other" was consuming significant
database time.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 6.42% of total activity.
Action
Investigate the cause for high "reliable message" waits. Refer to
Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .03 active sessions, 6.42% of total activity.
Action
Investigate the cause for high "reliable message" waits in Service
"mcmsdrac".
Recommendation 3: Application Analysis
Estimated benefit is .02 active sessions, 4.13% of total activity.
Action
Investigate the cause for high "reliable message" waits in Module "TOAD
9.7.2.5".
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
Finding 7: Unusual "Other" Wait Event
Impact is .03 active sessions, 6.29% of total activity.
Wait event "enq: PS - contention" in wait class "Other" was consuming
significant database time.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 6.29% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits. Refer to
Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .02 active sessions, 6.02% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits in Service
"mcmsdrac".
Recommendation 3: Application Analysis
Estimated benefit is .02 active sessions, 4.93% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits with
P1,P2,P3 ("name|mode, instance, slave ID") values "1347616774", "1" and
"3599" respectively.
Recommendation 4: Application Analysis
Estimated benefit is .01 active sessions, 2.74% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits in Module
"Inbox Reader_92.exe".
Recommendation 5: Application Analysis
Estimated benefit is .01 active sessions, 2.74% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits in Module
"TOAD 9.7.2.5".
Recommendation 6: Application Analysis
Estimated benefit is .01 active sessions, 1.37% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits with
P1,P2,P3 ("name|mode, instance, slave ID") values "1347616774", "1" and
"3598" respectively.
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
Finding 8: Hard Parse
Impact is .02 active sessions, 5.5% of total activity.
Hard parsing of SQL statements was consuming significant database time.
Hard parses due to cursor environment mismatch were not consuming significant
database time.
Hard parsing SQL statements that encountered parse errors was not consuming
significant database time.
Hard parses due to literal usage and cursor invalidation were not consuming
significant database time.
The Oracle instance memory (SGA and PGA) was adequately sized.
No recommendations are available.
Symptoms That Led to the Finding:
Contention for latches related to the shared pool was consuming
significant database time.
Impact is .09 active sessions, 22.42% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is .1 active sessions, 24.96% of total activity.
Finding 9: Soft Parse
Impact is .02 active sessions, 3.86% of total activity.
Soft parsing of SQL statements was consuming significant database time.
Recommendation 1: Application Analysis
Estimated benefit is .02 active sessions, 3.86% of total activity.
Action
Investigate application logic to keep open the frequently used cursors.
Note that cursors are closed by both cursor close calls and session
disconnects.
Recommendation 2: Database Configuration
Estimated benefit is .02 active sessions, 3.86% of total activity.
Action
Consider increasing the session cursor cache size by increasing the
value of parameter "session_cached_cursors".
Rationale
The value of parameter "session_cached_cursors" was "100" during the
analysis period.
Symptoms That Led to the Finding:
Contention for latches related to the shared pool was consuming
significant database time.
Impact is .09 active sessions, 22.42% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is .1 active sessions, 24.96% of total activity.
Finding 10: Unusual "Other" Wait Event
Impact is .01 active sessions, 3.75% of total activity.
Wait event "IPC send completion sync" in wait class "Other" was consuming
significant database time.
Recommendation 1: Application Analysis
Estimated benefit is .01 active sessions, 3.75% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits. Refer
to Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .01 active sessions, 3.75% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits with P1
("send count") value "1".
Recommendation 3: Application Analysis
Estimated benefit is .01 active sessions, 2.59% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits in
Service "mcmsdrac".
Recommendation 4: Application Analysis
Estimated benefit is .01 active sessions, 1.73% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits in
Module "TOAD 9.7.2.5".
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Additional Information
Miscellaneous Information
Wait class "Application" was not consuming significant database time.
Wait class "Commit" was not consuming significant database time.
Wait class "Configuration" was not consuming significant database time.
CPU was not a bottleneck for the instance.
Wait class "Network" was not consuming significant database time.
Wait class "User I/O" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
The database's maintenance windows were active during 100% of the analysis
period.
Please help.Hello experts...
Please do the needful... It's really very urgent.
Thanks,
Syed -
Install Oracle RAC 10g (10.2.0.1) on HP-UX B.11.31 U ia64 failed
Hi All
I am installing Oracle RAC 10g 10.2.0.1 on HP-UX B.11.31 U ia64 but can not complete
hosts file
#Public IPs
10.144.1.111 spgdb01
10.144.1.112 spgdb02
#Private IPs
10.144.2.2 spgdb01p
10.144.2.3 spgdb02p
#Virtual IPs
10.144.1.113 spgdb01v
10.144.1.114 spgdb02v
I do installation with runInstaller without error. It copy and link is ok. When I run root.sh then It cannot complete as following
Checking to see if Oracle CRS stack is already configured
Checking to see if any 9i GSD is up
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle/product/10.2.0' is not owned by root
WARNING: directory '/oracle/product' is not owned by root
WARNING: directory '/oracle' is not owned by root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 0: spgdb01 spgdb01p spgdb01
node 1: spgdb02 spgdb02p spgdb02
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
Now formatting voting device: /ora/crs/votedisk01
waitpid(-1, 0x7fffdf50, WUNTRACED) .................................................................................................... [sleeping]
Now formatting voting device: /oracle/oradata1/crs/votedisk02
Now formatting voting device: /oracle/oradata2/crs/votedisk03
Format of 3 voting devices complete.
Startup will be queued to init within 30 seconds.
====================
I have waited for 10 mins but still not complete
Additionally, log from runInstaller, I got
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2011-04-28_12-13-31AM. Please wait ...-bash-4.2$ Oracle Universal Installer, Version 10.2.0.1.0 Production
Copyright (C) 1999, 2005, Oracle. All rights reserved.
Private Interconnect : null
Private Interconnect : null
Private Interconnect : null
Private Interconnect : null
So, please help me fix this issue
Thank youI had this problem and resolved it by transporting the file to the installation server with the correct ftp datatype (binary).
On page 54 of the install guide (..Server\Oracle_Business_Intelligence\doc\doc\bi.1013\b31765.pdf) that comes with the installation files, there is an instruction to make sure that any ftp activity is done in binary.
This may not have occured with the license.xml file if you use a tool which offers the "feature" of automatic datatype recognition.
Hope this helps. -
APEX Application behaviour in a RAC setup
Hi
Caveat first: I'm pretty new to Oracle RAC and just looking into it as an option. We have an APEX application currently running in Oracle 11gR2 single node currently and are considering HA for this.
My question is: What would be the expected behaviour seen by a User of an APEX application, in the event of a node failure, when running with an OHS / RAC configuration? Will they get "transparent fail-over" and see nothing or will they see an error?
I appreciate I could post in the APEX forum, but feel that is probably more of a development forum and possibly someone here has had to look at things at this level.
I have read what I think may be the definitive reference for this:
http://www.oracle.com/technetwork/developer-tools/apex/learnmore/apex-rac-wp-133532.pdf
but while it covers most of what I want I don't believe I have found an answer to my question
This states:
"The Transparent Application Failover (TAF) feature of Oracle Net Services is a runtime failover for high-availability environments. It enables client applications to automatically reconnect to the database if the connection fails and, optionally, resume a SELECT statement that was in progress. The reconnection happens automatically from within the Oracle Call Interface (OCI) library. For applications that do insert, update or delete transactions, the application must trap the error when the failure occurs, rollback the transaction, and then resubmit. If the application is not written to be TAF aware, the session will get disconnected."
However (as I understand it) APEX runs in the database and would fail with the database, it isn't a typical "client application" connecting to Oracle via a TAF aware connection pool - it is essentially a large pl/sql package and TAF only covers SELECT statements not packages.
May be I'm over-reading this and it's simpler than that: APEX/Mod_plsql might just handle it?
- APEX User/HTTP session state is stored in database APEX: Understanding session state which is available on other nodes
- Mod_plsql in OHS can detect the error returning and reissues the request to good server and APEX on that instance can retrieve Users/HTTP state and process the request (APEX/RAC doc states mod_plsql can see an error from database and cleanup connection up and form a new connection, but not that it will retry the request for the client into other APEX/DB node).
I'm really just after a (transparent/non-transparent) statement based on experience, but an outline of how the components behave would be useful.
Thanks in advance
DaveHi
Any chance of getting that link outside of Metalink? - I'm trying to get our customer support id, but no luck at present.
I'm aware that APEX can run with RAC (as per the link I posted) - I'm really after next level info around behaviour in that environment.
Thanks
Dave -
Oracle Upgrade from Oracle RAC 9.2.0.6 to Oracle 10.2.0.4
Hi All,
Currently, we are running 4 node Oracle RAC environment with below mentioned configuration.
OS: Sun Solaris 5.9
Hardware: Sun E2900
Oracle Version: 9.2.0.6
Veritas Cluster Server: 4.1
We want to upgrade Oracle version to 10g, and currently analyzing the options to perform this. The current database size is 1TB appx and we want to spend minimum application shutdown time running on this database.
As part of upgrade, we also need to upgrade Veritas Cluster Server from 4.1 to 5.1 to support Oracle 10g. It would be great help,, if someone can pass some guideline to perform this task.
We are currently thinking about piecemeal approach, where we can upgrade each node individually and then put them back to cluster. There are some complexities involved, and its really high risk approach.
Thanks a lot in advance for help
Regards,
ManojOrcale 10g RAC requires you to install Oracle Clusterware. Oracle Supports running it along side a 3rd party clustering software. Not sure why you're so anxious to upgrade Veritas Clusterware when it will be trivial on the 10g db hosts.
Maybe you are looking for
-
Could not find agent library on the library path or in the local directory
Hi all, I'm trying to write a jvmti agent that write any information in a mysql db. I've written a simple agent that work correctly and now I'll try to insert the mysqlpp library in my agent: 1) I've added #include <mysql++.h> 2) and I've added mysql
-
What is the latest SP for BI7.0 AIX server
Hi, Now I am using SAP BI7.0 with SP14 and my Operating System is IBM AIX5.1 and 590. Now I am planning to up grade my SP14 to higher version. I am checking in Service market place I am able to see till SP18. But I am confused that till how many
-
On the Oracle XE BETA Trial License agreement
Hi all, I've got a question concerning the Oracle XE BETA Trial license (I'm referring to the document behind the "License Agreement" link in the "links" pane on the right top of the database home page) I understand that the Oracle XE Beta License gr
-
Forms 6i- Good variable naming convensions
Could you suggest any good naming conventions on Forms 6i
-
ASSIGN Component ... (field symbol stuff)
Hi, I want to validate certain fields (KNA1, BUKRS, KOART, FNAME) which are entered by the user in the transaction XK01. They are being captured in a structure, lets say X_LFA1. To validate this fields I am being asked to use field symbols using com