Running OEM for Other RAC Instances

Hi,
My RAC setup has 3 nodes and I know that more than one database exists (say Retail, Corporate, Private banking databases etc) on a single shared storage. The portlist.ini file from one of the node instance looks like below:
Enterprise Manager Console HTTP Port (a111) = 5500
Enterprise Manager Agent Port (a111) = 3938
Enterprise Manager Console HTTP Port (b111) = 1158
Enterprise Manager Agent Port (b111) = 3938
Enterprise Manager Console HTTP Port (c111) = 5501
Enterprise Manager Agent Port (c111) = 1831
Enterprise Manager Console HTTP Port (d111) = 5502
Enterprise Manager Agent Port (d111) = 1832
1. Is the above correct? Shouldn't there had been only one entry? Is there one entry each for every database?
2. The OEM is always defaulting to 1158. How can I make it run on other ports for all other databases?
Please help.
Thanks.

The concept behind dbconsole is, each database has its own dbconsole, which can only serve this particular database. Of course, in a RAC environment you have more than one instances per database, but dbconsole for database A does not know any other databases on the local or remote machines.
If you want to use dbconsole and you have x databases , you also have dboconsoles multiplied by x.
As an alternative you could use the old javabased EM, still available on the 10g client software. This EM can serve several databases.
Unlike dbconsole, which comes with the server software, Grid Control is independent of server/database installation. Mainly it consists of a Management Server and agents, which are installed on the target machines and collect all necessary informations for a centralized management.

Similar Messages

Running Multiple LIRC/[Other App] Instances in systemd Service

I'm trying to run multiple LIRC instances (one for each device) as a systemd service. This is what my service file looks like:
[Unit]
Description=LIRC Daemon
After=network.target
[Service]
Type=oneshot
EnvironmentFile=/etc/sysconfig/lirc
ExecStartPre=/bin/mkdir -p /run/lirc
ExecStartPre=/bin/rm -f /dev/lircd /dev/lircd1
ExecStartPre=/bin/rm -f /run/lirc/lircd /run/lirc/lircd1
ExecStartPre=/bin/ln -s /run/lirc/lircd /dev/lircd
ExecStartPre=/bin/ln -s /run/lirc/lircd1 /dev/lircd1
ExecStart=/usr/sbin/lircd --device=/dev/input/event12 --output=/run/lirc/lircd1 --pidfile=/run/lirc/lircd1.pid --driver=devinput --listen=8765 /etc/lirc/lircd.conf
ExecStart=/usr/sbin/lircd --device=/dev/input/event10 --output=/run/lirc/lircd --pidfile=/run/lirc/lircd.pid --driver=devinput --connect=localhost:8765 /etc/lirc/lircd.conf
ExecStopPost=/bin/rm -f /dev/lircd /dev/lircd1
ExecStopPost=/bin/rm -fR /run/lirc
[Install]
WantedBy=multi-user.target
I'm having some trouble with it as the service runs the first time but not properly. Restarting the service results in an Permissions error. I'm presuming because the socket is till running despite having manually deleted all LIRC sockets.
At the moment I'm thinking this is because service file excludes a PID assignment like:
PIDFile=/run/lirc/lircd.pid
Yet the two instances have their own PID assignments. What should I be putting or what's a better way to run two instances of LIRC as a service?
Last edited by archbawks (2013-04-13 16:58:59)

Anyone?

Local pfile for all RAC instances

DB Version: 10Gr2
OS : Solaris 5.10
2 node RAC
In one of our RAC DBs, we use local pfile for both of our instances.
If we had used a common (shared) spfile, we need to change parameter only once and it will be persistent across reboots.
Can you think of any other disadvantages of using local pfile? Are there any advantages in using pfile in RAC enviroments ?

Yes .. you are right ..
Oracle recommends that you manage parameters using an SPFILE. If you use client-side parameter files, then Oracle does not preserve parameter changes made for self-tuning after shutdown.
Check Oracle Documentation :
http://download.oracle.com/docs/cd/B19306_01/install.102/b14203/params.htm#CIHFCHGH
Regards
Rajesh

Windows error when running link for other application

When FF is not running and I click on a link in another application there is a error reported but FF loads and goes to the link. Example Outlook email link results in a Windows General failure error message.
Run Windows 7 sp1

Are you trying to run the app from a Command Prompt or fro Text Pad?
The error occurs when java can not find the class you tried to run. It is very likely a Classpath problem. It may work from the Command Prompt from the directory where the NumberConvert.class is located, enter "java -classpath . NumberConvert"

Can't we start a RAC instance without local pfile ?

DB version : 11.1.0.7.0
OS : Solaris 10 SPARC
--Info about the cluster
2 Node RAC
Both instances using spfile stored in ASM Diskgroup
Passwordless SSH configuration is not working currently (i gather ssh is not mandatory after installation)
We have 7 RAC DBs running in our 11.1 CRS.
After a reboot of our 2 Node RAC servers, in Node2, 4 instances didn't come up.
None of these down instances have a local pfile. The ones that are up in Node2 have it.
So, When i manually tried to bring up the ones that are down
$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 - Production on Fri Mar 2 21:28:22 2012
Copyright (c) 1982, 2008, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/11.1/db_1/dbs/initorcl2.ora'
SQL> exit
Disconnected-- Tried directly with spfile (No such option in oracle, but still tried ! )
SQL> startup spfile='+ORCL_DATA01/ORCL/spfileorcl.ora'
SP2-0714: invalid combination of STARTUP options-- Tried from Node1
-- This might have failed because ssh is not working currently. Not sure
$ srvctl start instance -d orcl -i orcl2
PRKP-1001 : Error starting instance orcl2 on node manhclust199
CRS-0215: Could not start resource 'ora.orcl.orcl2.inst'.
$ srvctl start database -d orcl
PRKP-1001 : Error starting instance orcl2 on node manhclust199
CRS-0215: Could not start resource 'ora.orcl.orcl2.inst'.Finally I had to create a local pfile on node2 for each instances in OH/dbs
The content of each local pfile is like
$ cat initorcl2.ora
spfile='+ORCL_DATA01/orcl/spfileorcl.ora'Now the startup worked without any issue for all 4 instances
SQL> startup
ORACLE instance started.
Total System Global Area 2137886720 bytes
Fixed Size                  2155464 bytes
Variable Size            1593838648 bytes
Database Buffers          536870912 bytes
Redo Buffers                5021696 bytes
Database mounted.
Database opened.So, local pfile is mandatory for all RAC instances ?

Hi,
My question is whether this instance specific local pfile must be there in OH/dbs directory ?YES
Didn't get a clear answer yet.
It seems that it is mandatory. And this is most probably the reason these instances didn't come up after the reboot of this node.Startup Sequence:
Startup of Cluster:
Using "srvctl" to start your database/instance:
Srvctl read the OCR and check the parameter SPFILE from your database registration:
If parameter SPFILE in OCR have a value the srvctl will use this value to start your database instance.
If parameter SPFILE in OCR don't have a value (i.e null) Oracle will try find the file SPFILE<ORACLE_SID>.ora on $ORACLE_HOME/dbs
If the file SPFILE<ORACLE_SID>.ora does not exists, Oracle will try use the file init<ORACLE_SID>.ora
If init<ORACLE_SID>.ora does not exist... the startup will fail.
From ORACLE_HOME of your database execute command above to get value of Spfile parameter on OCR:
srvctl config database -d <DB_NAME> |grep SpfileStartup manually from SHELL using SQLPLUS:
Using SQLPLUS to start a database Oracle always first use SPFILE<ORACLE_SID>.ora stored on $ORACLE_HOME/dbs/ if this file does not exist he will try use INIT<ORACLE_SID>.ora to start Database Instance.
If your spfile is stored on ASM you must create a INITFILE pointing to SPFILE stored on ASM.
e.g From instance 1
cat /oracle/oracle/product/11.2.0/db_1/dbs/initorcl1.ora
SPFILE='+DATA/ORCL/spfileorcl.ora'Regards,
Levi Pereira

How to open oracle listner for three database instances

please help me with following questions
I have 3 database instances
orcl
JIP02T
JIP04T
when i start the oracle database, does it start all 3 instances? when i start the oracle listner, does it start listner for all 3 instances?
when i echo $ORACLE_SID
[ora112@localhost admin]$ echo $ORACLE_SID
orcl
i can only see orcl and oracle listner is started only for one database instance.i can connect to only one database instance usign sql developer how can i start oracle database listner for other database instances as well

ShishirTekadeR wrote:
. oraenv
Note that is "dot" SPACE oraenv
You will be prompted for the value of ORACLE_SID (one of the databases). It looks like this:
oracle:orcl$ . oraenv
ORACLE_SID = [orcl] ? dwsbx
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle
oracle:dwsbx$
HIGHLY APPRECIATED U R IMP Tips!!!!!
But it’s very difficult to remember & manually enter oracle SID & HOME details every time after run (. Oraenv)
.profile creation is simple method and no need to remember SID & HOME details.(for multiple DB in one server)You don't have to "remember & manually enter oracle SID & HOME details every time after run " oraenv sets all those details for you. All you have to remember is the name of the instance you want to work with. The same thing you'd have to remember if you if you created (as you suggested) three different "profiles" - actually three different shell scripts, each to set the environment for a specific instance.
Yes, I do place "default" settings in .bash_profile. But with multiple databases the "default" settings are often not going to be what you need so you'll still have to run some process to change them. Either a home-grown script or oraenv. And either way you will have to remember the names of the instances you have to chose from. So in general I see no advantage of reinventing the wheel.
>
>
>
As per user information
He is using different port number for different instanceWhich accomplishes absolutely nothing except extra configuration headaches. It accomplishes nothing for security. It accomplishes nothing for availability.
And also it is very simple method to manage three different DB in three different listener entry (for DB maintenance activity.)Not as easy as managing one default listener. By far the most frequent listener configuration issue I see on this forum is when people try to configure one listener per database.
Single listener in multiple instance (if listener down all DB connection will affect)And why would the listener be down? I've been working full time with Oracle since version 7.3 - somewhere around 1995. I've NEVER seen a listener fall over and die. I have seen many people on this forum whose db connections failed because they were trying (unsucessfully) to configure multiple listeners.
All of my servers are running multiple databases. I have one test server that is running close to a dozen separate databases. One listener. I've NEVER had a situation where I thought to myself, "Gee, it would really help if I had created a separate server for each database"
>
>
Best Regards,
Shishir Tekade.
My Blog: http://shishirtekade.blogspot.com
Edited by: ShishirTekadeR on Dec 27, 2012 10:29 PM
Edited by: ShishirTekadeR on Dec 27, 2012 10:30 PM
Edited by: ShishirTekadeR on Dec 27, 2012 10:31 PM

CREATING A SINGLE INSTANCE PHYSICAL STANDBY FOR A RAC PRIMARY

Hi
Creating a single instance physical standby database for a RAC Primary.
Getting this error.
sql statement: alter database mount standby database
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 01/17/2008 23:05:38
RMAN-03015: error occurred in stored script Memory Script
RMAN-03009: failure of sql command on clone_default channel at 01/17/2008 23:05:38
RMAN-11003: failure during parse/execution of SQL statement: alter database mount standby database
ORA-01103: database name 'PROD' in control file is not 'DPROD'
Any help on this.
Regards
Satish

The problem here is probably with your standby init.ora file.
When you create a standby database, the db_name parameter must NOT change. It has to match the primary database. So in your case, db_name ='PROD' and your db_unique_name='DPROD'...
-peter

Creating a RAC standby database for a single instance database

Dear All,
I have a task of migrating a 500GB single instance database to a two-node RAC database with a little downtime at hand. My migration strategy is to:
1) Create a RAC physical standby for the Single Instance database
2) Switchover to RAC standby.
Primary and Standby OS and DB configurations:
OS: Windows Server EE 2003 (64-bit)
DB: Oracle 10g Database Release 2 (10.2.0.4)
Oracle 10g Clusterware Release 2 (10.2.0.4)
To create a RAC standby, I will:
a) Install Clusterware (10.2.0.1)
b) Install Database (10.2.0.1)
c) Patch both Clusterware and Database (10.2.0.4)
d) Create ASM instance for both the nodes (+ASM1 & +ASM2)
e) create standby controlfile on primary
f) Move standby controlfile, RMAN backup of primary, pfile, listener.ora, tnsnames.ora, password file to standby host-1
g) make necessary changes to the pfile on standby host-1 like cluster_database, instance_name, thread, ...
h) mount standby database and restore backup
Kindly validate my steps and if there already exists such a document then please do provide me with a link.
Regards

Please refer to MAA whitet paper :
[http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10g_RACPrimaryRACPhysicalStandby.pdf]
[MAA website|http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm]

Cleint tnsnames.ora for connecting to RAC instance

Dear all,
DB:10.2.0.1
o/s:redhat 5.6
2node rac cluster
I am getting ERROR: ORA-12170: TNS:Connect timeout occurred when trying to connect to rac instance.the timeout frequency is very less once in 10 times.
cleint tnsname.ora
RACTEST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.25)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.27)(PORT = 1521))
(FAILOVER= TRUE)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RACTEST)
Regards

Try to connect straight to the nodes. One node at the time. You can add new tnsnames entries for all RAC nodes so you can connect only one node not the whole RAC.
example:
RACTEST1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.25)(PORT = 1521))
(CONNECT_DATA =
(SERVER = dedicated)
(SERVICE_NAME = RACTEST)
(INSTANCE_NAME = RACTEST1)
RACTEST2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.27)(PORT = 1521))
(CONNECT_DATA =
(SERVER = dedicated)
(SERVICE_NAME = RACTEST)
(INSTANCE_NAME = RACTEST2)
If both nodes are working then problem is probably in network / firewall.
Usually this kind of problems comes because one node's database instance is somehow broken. And if that is the problem correction is restart failing instance.
Edited by: Simo Kemppinen on Jul 15, 2012 12:58 PM
Edited by: Simo Kemppinen on Jul 15, 2012 12:59 PM

Mac is freezing no programs running yet no trouble for other user

My mac keeps freezing yet no trouble for other user. No program's run without 5 mins of coloured wheel rotating. Also time machine not operating properly as computer is not recognising it. Apple support centre not much help as it took them 10 mins just to get my details straight! Anyone with similar problem and successful solution?
Thanks.

Tried Mac HD utility first aid but advised it can't repair disk.
It's possible you have directory damage. You will need to get that repaired asap by using/purchasing a stronger 3rd party utility. The more you use the computer, the worse it will get.
Utilities are best run from the CDs to avoid system interference & false readings/reports. You must use the versions that are compatible w/your OS system & keep the utilities updated to avoid damaging/harming/trashing your system. You need to make your own decision on which to purchase. Read up on them on their websites because each do something a little different.
DiskWarrior (Highly Recommended for Directory Damage Repair)
Techtool Pro
Drive Genius

HT204350 I am trying to transfer files from MacBook running OSX 10.7.5 toIMac running OSX 10.9.1. I verified all software updates installed. Both connected directly by ethernet cable. On the IMac I get "Looking for other sources". On MacBook get "look

I am trying to transfer files from my MacBook (source) running OSX 10.7.5 to my IMAC (target) running OSX 10.9.1. Machines connected by ethernet cable. On Target machine I get "Looking for other sources" and on source machine I get "Looking for other computers". Both have clockwise spinning wheel.
Looking for advise.
Thanks
Luap N

If you are connecting the two computers directly to each other with an ethernet cable, you need a special "crossover" cable to swap the send/receive pins so the data sent from one computer arrives on the "receive" pins on the other end of the cable, not on the "send" pins that you get with a straight cable.
Incorrect, you do not need a special crossover cable. Any cable will do.

I am trying to detect signal peaks (derivatives of the peaks both +ve and -ve peaks). simultaneously 2 peaks detectors are running, 1 for +ve and other for -ve peaks.

i am trying to detect signal peaks (derivatives of the peaks both +ve and -ve peaks). simultaneously 2 peaks detectors are running, 1 for +ve and other for -ve peaks. when a peaks get detected it would trigger a pulse of approx 1 sec duration and at the same time it would disable the other peak detector for 1 sec time duration. after the time period, again both the peak detectors would be enabled. i have been able to detect both peaks but still not able to control the detectors block the way mentioned before.
the code and the data are attached.
Attachments:
eog peak detection.vi ‏192 KB

I'm not suggesting any of this as a model, and your needs are different than mine.
1) Go to Preferences -> Audio. Under "Default Audio Tracks", Set "Mono Media: Mono". Set "Stereo Media: Mono" (Choices from the pull-down for both are Use File, Mono, Stereo, 5.1, Adaptive)
You can change these, but I don't. I am using both mono and stereo tracks. I set up a custom sequence with the particular combination of stereo and mono tracks I want.
2) Set audio for clips by selecting all clips and going to Modify Clip -> Audio Channels Tab. Under "Number of Audio Tracks" there should be a "1". Under Channel Format it should read "Stereo".
I am not sending the audio to anyone else to edit further. I am doing a final stereo mix. If camera one is set up to record 2 mikes as a stereo track, then I leave that unchanged and put it onto a stereo track. If camera two is set up to record one mike onto channel 1 as ambient sound and channel 2 is a mono feed from a sound board, then I modify those clips as Preset Mono, 2 audio tracks, Channel format Mono, and that puts the left source channel (or whichever I choose) onto Audio track 1, channel 1 (of one), and right source channel onto audio track 2, channel 1.
4) Delete all the existing audio tracks in the sequence and replace them all with mono tracks, so all tracks in the sequence are mono.
I modify the tracks while creating the sequence, and ultimately use a custom sequence.
Under my Effects Controls panel there is only Channel Volume, and nowhere to adjust the pan.
I'm using the mixer, not the effects control panel.

Oracle RAC instance related

We have Oracle Databases 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production running on Linux x86 64-bit
It is a two instance RAC running on servers, let's say - node1 and node2 . We are using ASM
Node 1 has an ASM instance ASM1 and Node 2 has an ASM instance ASM2.
There are 3 11g rdbms databases running on these nodes.
Instances db11,db21,db31 are running on node 1 and corresponding RAC instances db12,db22,db32 are running on node 2
The listeners are configured exactly the same on both nodes.
On Node 2, when I do
[oracle@node2 admin]$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 19-NOV-2010 14:34:34
Copyright (c) 1991, 2009, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                15-NOV-2010 13:33:49
Uptime                    4 days 1 hr. 0 min. 44 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/oracle/product/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/node2/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.7.42)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.7.52)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "db1" has 2 instance(s).
Instance "db11", status READY, has 1 handler(s) for this service...
Instance "db12", status READY, has 2 handler(s) for this service...
Service "db2" has 2 instance(s).
Instance "db21", status READY, has 1 handler(s) for this service...
Instance "db22", status READY, has 2 handler(s) for this service...
Service "db3" has 2 instance(s).
Instance "db31", status READY, has 1 handler(s) for this service...
Instance "db32", status READY, has 2 handler(s) for this service...
The command completed successfullyThe above looks good which is what should be the case.
Now, if I try doing the same on node 1 (and this is where I am concerned)
[oracle@node1 admin]$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 19-NOV-2010 14:41:45
Copyright (c) 1991, 2009, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                19-NOV-2010 03:20:44
Uptime                    0 days 11 hr. 21 min. 1 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/oracle/product/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/node1/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.7.41)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.7.51)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "db1" has 1 instance(s).
Instance "db11", status READY, has 2 handler(s) for this service...
Service "db2" has 1 instance(s).
Instance "db21", status READY, has 2 handler(s) for this service...
Service "db3" has 1 instance(s).
Instance "db31", status READY, has 2 handler(s) for this service...
The command completed successfullyThe node 1 does not seem to report the fact that each of these 3 databases have 2 instances and also does not list its other instances besides the ones running on it. Any ideas or suggestions as to where to look?

One problem is here
The listeners are configured exactly the same on both nodes.This is incorrect, as the listeners need to have different names.
They need to have different names as listener_node1 is the remote_listener for node 2 and vice versa.
The correct set up is:
The listener name is node dependent.
The listener definition
listener_<node>=(host=)(protocol=)(port=)
is included in tnsnames.ora
the remote_listener parameter is set to the listener of the other node.
Using hardcoded IPs in listener.ora and tnsnames.ora is a bad idea.
Not sure why you don't ask this question in the RAC forum.
Sybrand Bakker
Senior Oracle DBA

How to determine which RAC-instance the appl. is logged onto?

Dear all,
I need to have my application server determine which RAC-
instance is currently active (logged onto). I have a
tnsnames.ora file with a primary-, and secondary RAC-
instance configured, and Failover/Failback between the
instances work fine. However, I would be interested in
determining which instance I am curently using.
Does the Oracle Net Protocol have support for letting me
"read" this out, or...?
Thanks.
Regards, Eldor R.

Thank you for the prompt reply.
Is there, in the Oracle Net Protocol, available
function(s) for reading out this information
directly without "parsing" the trace file?
I would like to read out this information from my
application run-time.
Thanks.

RAC instance won't start: ORA-00443: background process "VKRM" did not star

I've logged a SR with Oracle, but while waiting for response from them...I'm stumped I can't find more out here about this article.
On Oracle knowledgebase...any search for VKRM, gives about the same 3 articles relating to RDA (Remote Diagnostic Assistant).
Not sure what went on here.
I have a 5 node RAC cluster. All other instances seem to be running just fine.
On one instance...some applications were getting an error like:
ORA-01033: ORACLE initialization or shutdown in progress
I looked in GRID...and it indicated that only two of the 5 nodes had this instance running...which was strange in that with srvctl, it showed ll 5 up and running:
[oracle@server2 bin]$ ./srvctl status database -d INSTANCE
Instance INSTANCE1 is running on node server1
Instance INSTANCE2 is running on node server2
Instance INSTANCE3 is running on node server3
Instance INSTNANCE4 is running on node server4
Instance INSTANCE5 is running on node server5
Anyway, thought I'd poke around. I started with trying to get srvctl to stop instance #2...in GRID it seemed that instance 2,4 and 5 weren't working.
srvctl stop instance -D INSTANCE -i INSTANCE2
This just hung...
I thought I'd cycle all the nodes...so, did a ctl-c out of that one, and did:
[oracle@server2 bin]$ ./srvctl stop database -d INSTANCE -o abort
PRCD-1124 : Failed to stop database INSTANCE and its services
PRCR-1065 : Failed to stop resource (((((NAME STARTS_WITH ora.instance.) && (NAME ENDS_WITH .svc)) && (TYPE == ora.service.type)) && ((STATE != OFFLINE) || (TARGET != OFFLINE))) || (((NAME == ora.instance.db) && (TYPE == ora.database.type)) && (STATE != OFFLINE)))
CRS-2675: Stop of 'ora.instance.db' on 'server5' failed
CRS-2675: Stop of 'ora.instance.db' on 'server4' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
The ORA-01034 messages repeats a number of times...and the one with server4 and server5 repeated again too.
I also got this:
CRS-2680: Clean of 'ora.instance.db' on 'server2' failed
CRS-2675: Stop of 'ora.instance.db' on 'server5' failed
CRS-2675: Stop of 'ora.instance.db' on 'server4' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
I get similar messages when I try to restart.
clusters seem to be up, and other instances seem to be ok
Looking at alert log...I found some strangeness in the traces
Starting background process VKRM
Errors in file /u01/app/oracle/diag/rdbms/instance/INSTANCE2/trace/INSTANCE2_dbrm_26982.trc:
ORA-00443: background process "VKRM" did not start
Errors in file /u01/app/oracle/diag/rdbms/instance/INSTANCE2/trace/INSTANCE2_ora_27467.trc:
ORA-00450: background process '' did not start
Errors in file /u01/app/oracle/diag/rdbms/instance/INSTANCE2/trace/INSTANCE2_ora_27467.trc:
ORA-00450: background process '' did not start
Error 450 happened during db open, shutting down database
USER (ospid: 27467): terminating the instance due to error 450
LGWR waiting for instance termination
Instance terminated by USER, pid = 27467
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (27467) as a result of ORA-1092
Looking at the trace listed above:
2011-05-09 12:17:18.305726 :84271119:db_trace:ksb.c@2157:ksbs1p_real(): [10254:6:464] KSBS1P: process DBRM trying to start background VKRM
2011-05-09 12:17:18.305729 :8427111A:db_trace:ksb.c@2220:ksbs1p_real(): [10254:6:464] KSBS1P: process DBRM obtained PR enqueue to start background VK
RM
2011-05-09 12:17:18.306021 :8427111D:db_trace:ksb.c@2354:ksbs1p_real(): [10254:6:464] KSBS1P: creation error posted OER(1089)
2011-05-09 12:17:18.306029 :8427111F:db_trace:ksb.c@2424:ksbs1p_real(): [10254:6:464] KSBS1P: out of loop: process did not start
Trace Bucket Dump End: default bucket for process 6 (osid: 26982, DBRM)
ORA-00443: background process "VKRM" did not start
kskdbrmpa: reply error 450
Any ideas? Again...I can't seem to find much of ANY information searching out there for the VKRM background process not starting...
Thanks in advance,
cayenne
Edited by: cayenne on May 9, 2011 11:34 AM

Anyone? Anyone? Bueller?
Ok..have been on phone with Oracle support, and have them stumped so far.
I've checked..other instances are running (except one other that failed to cleanly shut down with srvctl and same error messages).
I've check...ASM is running on all 5 nodes. I've used crsctl to check CRS on all nodes..clustering seems ok.
Memory while somewhat high...should have enough room...the system has never complained before, and this has been up on this config for over a year.
I was able on node one...to fire up the first nodes instance using the pfile there...it came up. I started it restrict..and promptly ran a datapump export, and then shut back down.
While Oracle support is going through logs and trace files I sent...I've got another 11Gr2 environment (3 node RAC) I've been using as a test environment...and am recreated the instance there temporarily to allow my developers to test and get past the upcoming deadline.
But, I'm still puzzled as to the solution on the main cluster. This seem to be normal, except these two instances....
any ideas on where and what to look for?
cayenne

Running OEM for Other RAC Instances

Similar Messages

Maybe you are looking for